Robots.txt example for WordPress

Currently I’m doing some research for BloggingSupport.com, a product that will manage people’s websites that run WordPress. One of the interesting problems I’ve seen from clients sites, and my own, is that content that can easily be blocked by a robots.txt file, simply isn’t.

This can lead to a whole host of problems including:

  • sensitive content being indexed that shouldn’t
  • back links given from your site to plugin authors
  • negative SEO attacks
    • duplicate content from search pages issues
    • high number of indexed pages with thin content

It’s best practice to have such a file and I recommend you add one.

robots.txt example

Note : This example assumes that your blog is in the root folder of your domain.


User-agent: *
Disallow: /wp-login.php
Disallow: /wp-admin/
Disallow: /wp-content/plugins/ # prevents backlinks in plugin folders
Disallow: /wp-includes/
Disallow: /search/ # prevents search queries being indexed
Disallow: /*?s

You can see our robots.txt file here https://www.blackdog.ie/robots.txt.

To install a robots.txt, simply put a file with this name in the root folder of your domain.

Google indexing robots.txt

If you do a search on google from [site:mydomain.com] and your robots.txt or sitemape.xml file appear high in the results, you may want to discourage google from indexing those pages too. The difference here is that you want them to be read & process by google, just not indexed to appear in the search results.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.