Hiya! robot.txt is the file which search engines read and decide that what kind of data they have to index from your site in the search engines also exclude those which are not. This file contains the HTML or XML sitemaps URLs which then indicate the search engines about the whole structure of your site. I was trying to find robot.txt file in WordPress, and edit that file anyway to add the sitemaps URLs in that file for better crawlment!

It is the most necessary file for the search engines crawlers in order to get the new data from your site. From my personal experience, I used every way to edit this file but I couldn’t. Nevertheless, I didn’t give up and found a pretty good yet awesome WordPress plugin which enable me to edit the robot.txt file on demand.

WP Robots.txt in WordPress

Basically, it’s a WordPress plugin which let us edit the robots.txt file on demand. Just follow these steps to edit the robot.txt file and add the sitemap urls in it:

1. Download, install and activate wp robots txt plugin.

2. Go to Settings > Reading

3. Now, simply add the sitemap URLs (HTML or XML) like this:

Sitemap: https://softstribe.com/sitemap.xml
Sitemap: https://softstribe.com/html-sitemap

Note: to generate the XML or HTML sitemaps read this article.