Google Crawling and Indexing



To understand Crawling and indexing you have to first understand search engine bot. It is also called search engine crawler or spider. For different search engines the bots are differently pronounced:


Google - Googlebot

Yahoo - Yahoobot
bing - bingbot
ask - askbot


What is Googlebot or Google Crawler?

Googlebot (abbreviated Google Crawler or Google Spider) is a search bot software that Google sends to websites to gather information about documents on world wide web (www).

What is Crawling?

Crawling is the process where the search bot or crawler goes around from website to website, finding new and updated information to report it back to Google. The search engine crawler finds what to crawl using links. 

Basically there are three steps of a web crawling procedure:


1. First search bot starts crawling the pages of your website


2. Second these bots continue crawling for indexing the updated information on web.


3. Third the search bots also visit the links (URLs) within your webpage  and decide to crawl new web page.


When any search bot does not find the web page or link is broken, it will eventually deleted that web page from search engine's index.


What is Indexing?

Indexing is the process of collecting information by search crawler from it's crawling activities. Once the web page is crawled by crawlers they added it to search engine's index.

The results appear in Google search after crawler indexed your website. Searchbot processes the whole content within a web page where this content is located such as title tags, ALT attributes, heading tags and hyperlinks.