Updated – August 2020
Search engines are available to help the internet users to discover and learn the content present anywhere on the internet.
When a user searches anything on search engines, he gets a number of results. The top result page contains only 10 results and most of the users visit only these.
So, everybody wants to get his content published within these first 10 results. To get the position there, first, your content needs to be visible to the targeted search engine. A simple logic:” If your site is not visible to search engines, then you are also not visible to the internet users.”
First of all, you need to understand, what exactly a search engine does?
There are three main functions of any search engine:
- Crawling the content.
- Indexing the content.
- Ranking the content.
Crawling the Content: Crawling is the first process in which search engine gives commands to the team of robots (also known as crawlers or spiders) to discover new and updated content on the internet. Content can be a webpage, an image, a video, or even a PDF, etc.
The crawler starts fetching the web page and also follows the links available on that webpage to find some new URLs.
It is also possible to restrict the crawlers to crawl some web pages, which can be done by robots.txt file. Read more about a robots.txt file: https://moz.com/learn/seo/robotstxt
Indexing the Content: When Search engines find something new in the crawling process then they store that new information into their databases. The process of storing the information in the database is known as indexing.
Ranking the Content: When a user performs a search, the search engine looks in its database for the most relevant content and finds a huge volume of results. Then search engine orders those results to solve the user’s query.
This ordering of results by their relevancy is known as ranking.In general, if the search engine finds a website more relevant, it gets a higher ranking on search engine results pages.
If you have blocked the crawling of some content through robots.txt file, the search engine will not show your content even if it was the most relevant result for the user’s query. Your site must be indexed by search engines. To be indexed it has to be crawled.
Crawling: Instruct search engines how to crawl your site
As mentioned above, to get a huge amount of organic traffic on your site, it must be indexed. For indexing, it has to be crawled.
So, before moving towards crawling your site, you must know how to check whether your site is indexed or not. If a webpage is already indexed, there is no meaning in recrawling.
To check the index status of your website, you can simply head to Google and type “site:yourdomain.com” in the search bar. Google will show you all the pages which are indexed.
If you don’t see your domain in the search results, then there are reasons for it-
- Your site is very new and has not been crawled by the GoogleBot.
- There is no link coming towards your website. These incoming links are also known as “backlinks”. To learn about the backlinks, click: https://www.shoutmeloud.com/backlink.html
- Your site is disallowed by robots.txt file.
- Your site has been penalized by Google. Read all Google penalties here: https://www.seoliquido.com/what-are-algorithmic-and-manual-penalties-of-google/
From the above reasons, if your site is not indexed because of zero backlinks, even then you can manually apply to crawl your site from “search console” by submitting your URL in “fetch as Google” option. There’s no guarantee Google’ll include your submitted URL in the index database, but it’s worth a try!
Even if everything was perfect, you can choose the above method to crawl your site when it’s new.
If you are the site owner, you have the full control to instruct the search engine on how to crawl your website.
You can instruct the Googlebot through robots.txt, meta tag, sitemap.xml, or Google Webmasters Tool (Search Console).
Indexing: How do search engines understand and remember a site?
Once your site has been crawled, the next step is to make sure it is indexed as well.
Yes, you thought it right. Just because your site is crawled by a search engine doesn’t necessarily mean that it will be definitely stored in their index.
In the previous section on crawling, we discussed how search engines find your web page. The index is where your all discovered pages are stored.
When a crawler finds a page, the search engine renders it. In the process of doing so, the search engine analyzes the content of that particular page. All of that information is stored in the index.
Maybe some of you have a question, Can I see, how a crawler sees my pages?
Yes! The cached version of the webpage will show you how did last time Googlebot crawl it.
The frequency of crawling and caching the websites are different for different websites. More established, well-known sites that post more frequently will be crawled more frequently than the less-famous websites.
You can see what your cached version of a page looks like by clicking the green drop-down arrow next to the URL in the SERP and choosing “Cached”.
After opening the cached version, you can also view the text-only version of your site to check whether all of your important contents are being crawled and cached, or not.
Google can also remove the indexed page after indexing it successfully; there are some reasons, which force Google to remove the already indexed pages:
- If your URL returns errors like 404 or 500.
- If you add a “noindex” tag to the page.
- If URL has been manually or algorithmically penalized.
You need to solve the above-written issues to reindex your webpage.
Ranking: How do search engines distribute priorities to all the websites?
Once you get indexed in the search engine database, the only work you have left is to rank your content.
Have you ever thought: how does Google ensure that when a user types a query in the search bar, he gets relevant results in return?
This process of providing the most relevant results is known as ranking, or the ordering of results from the most relevant to the least relevant for a particular search query.
To calculate relevancy, search engines use some secret algorithms, a formula through which stored information is retrieved and ordered in the most meaningful way.
These algorithms have faced many changes over the years in order to improve the quality of search results.
Google, for example, makes algorithm updates almost every day. Some of these updates are minor, but others are core/broad algorithm updates deployed to fight with some particular issues, like Penguin to fight with link spamming.
Why is the algorithm changed so often? Google does not always reveal exactly what they want. One thing is clear- Google’s aim is just to improve overall search result quality.
That’s why, in reply to algorithm update questions, Google answers with the lines:” We’re making quality updates all the time.”
This clearly means that, if your site started suffering just after an algorithm adjustment, you should compare it against Google’s Quality Guidelines. Read all quality guidelines here: https://support.google.com/webmasters/answer/35769?hl=en
Now, you may be thinking about the techniques through which you can rank your site well.
So, SEO is the answer to all of your questions.
SEO is basically related to links and content. There are two types of links in the world of internet, backlinks and internal links.
Backlinks or “inbound links” are those links that are coming from other websites to your page, while internal links are the links on your own site that point to your other pages of the same domain.
Links have played a big role in the history of SEO. At the beginning of digital marketing, search engines used to calculate the backlinks for any given site, to determine the worthiness of that website against the searched keyword.
Backlinks work as real-life Word-Of-Mouth referrals. Let’s take a hypothetical Tea shop, Tony’s Tea, as an example:
Referrals from others = a very good sign of authority
Example: Many different people claim that Tony’s Tea is the best in the city.
Referrals from yourself = biased, so not considered as a good sign of authority
Example: Tony claims that Tony’s Tea is the best in the city.
Referrals from irrelevant or low-quality sources = not a good sign of authority and can even create a serious issue of penalizing.
Example: Tony paid to people who have never visited his Tea shop to tell others how good it is.
No referrals = unclear authority
Example: Tony’s Tea might be good, but you are unable to find anyone who has an opinion or review so you can’t be sure.
After backlinks, content is another worthy material in the eyes of search engines.
All the search engines are answer machines, which provide answers to all the queries of search users.
So, a search engine can answer only in the form of content. Content can be an image, a video, a document, or plain text.
Whenever you post the content on your site, you have to be sure that the particular content you are going to post is original and has no plagiarism.
Nor Google no users like to read duplicate content.
According to Google, there are around 200 factors that are considered in the calculation of the ranking of search results. But still, there are 3 main factors deciding the ranking-Content, Backlinks, and of course, RankBrain.
Now the question arises, what is RankBrain?
In simple words, RankBrain is a machine learning code of Google’s core algorithm.
Machine learning is just computer code or program which is improving its predictions continuously through new observations and training data.
As it is always learning, search results quality is also improving over time.
For example, if RankBrain notices a URL which has a low ranking but provides a much better result to users than the URLs having the higher ranking, then it is confirmed that RankBrain will adjust those results automatically.
Google Knowledge Graph
Well, we have a brain. However, the brain memorizes thorough neural synapsis. Accordingly, Google’s brain runs on the Google knowledge graph.
This is the way Google to process and connect information. It processes massive data to present results in a meaningful way. These results can take a range of formats- snippets, images, or knowledge panels.
The panel is a result of the Google’s thinking, memory, or opinion. That’s the highly displayed and comprehensive result you get for certain queries.
In order to appear in this kind of snippet, you have to become an authority within your niche. After writing great, SEO optimized content, you have higher chances of getting featured in the panels.