Last time we have shown you how to index your website to search engines, this time we are going to talk about how to avoid pages from crawling. Search engines browses millions websites every day to get the most updated information using web crawler, an internet bot that do the browsing works automatically. There are other terms to describe these little internet bot such as web spider, automatic indexer, ant, and web scutter.
The purpose of doing web crawling is to update the web content in their database, however, to avoid any undesirable content such as in the search indexes, you can instruct the web spiders not to crawl them by putting the urls in the robots.txt file and store it in the root directory of the domain. As the robots.txt is the first file web spiders will look into, if they found something that the site owner doesn’t want to be crawled, the web spiders will skip them. Please note that not to over disable too many urls or it will discourage search engines come to your site. Thanks for reading and see you next time.