Robots.txt example for WordPress
Currently I’m doing some research for BloggingSupport.com, a product that will manage people’s websites that run WordPress. One of the interesting problems I’ve seen from clients sites, and my own, is that content that can easily be blocked by a robots.txt file, simply isn’t.
This can lead to a whole host of problems including:
- sensitive content being indexed that shouldn’t
- back links given from your site to plugin authors
- negative SEO attacks
- duplicate content from search pages issues
- high number of indexed pages with thin content
It’s best practice to have such a file and I recommend you add one.
Note : This example assumes that your blog is in the root folder of your domain.
Disallow: /wp-content/plugins/ # prevents backlinks in plugin folders
Disallow: /search/ # prevents search queries being indexed
You can see our robots.txt file here https://www.blackdog.ie/robots.txt.
To install a robots.txt, simply put a file with this name in the root folder of your domain.
Google indexing robots.txt
If you do a search on google from [site:mydomain.com] and your robots.txt or sitemape.xml file appear high in the results, you may want to discourage google from indexing those pages too. The difference here is that you want them to be read & process by google, just not indexed to appear in the search results.