Google crawls your site just after reading the most essential files like robots.txt and sitemap.xml/sitemap.html from your server. Let me clear what these files do? The sitemaps contains the whole data of your site including links of posts, pages, tags, categories, authors, etc, can be said the whole structure of your site in only one file either in HTML or XML format. Robots.txt file tells the search engines which pages they have to crawl and also exclude those which don’t. This file usually contains sitemaps URLs, for WordPress it looks like:

Adding sitemap (XML and HTML) urls in robot.txt

More info on how to edit robots.txt file in WordPress, read this article!

When Google Crawls My WordPress Site

Use Google Bot Bling to Know When Google Crawls your WordPress Site!!!

Well, Google Bot Bling is a WordPress plugin which describe complete information about when the Googlebot came on your WordPress site and what did the bot request from your site, and crawled. Until this plugin is in early development and it is designed well to help WordPress site owners when Google indexes their sites by sending emails from time to time.

How it Works?

There is a plugin settings page where you have to add your email and specify the time interval of sending emails about the Googlebot. To add the email and set the time interval follow these steps:

1. Download, install, and activate the plugin Google Bot Bling

2. Go to Settings > Google Bot Bling

Google Bot Bling Settings Page

3. What about secret file name? The secret file tells you about the requested data from your site by a bot on daily basis.

4. After setting up the plugin you’ll be receiving the emails from Google Bot Bling whenever it found the new crawled pages. See the email sent by this plugin:

Email from Google Bot Bling

Enjoy blogging with WordPress! 😀