I was thinking that with a test WordPress website, it would be a good idea to make sure it wasn’t indexed by the search engines. So, I did some research and here are my steps:
- Login into WordPress admin
- Settings –> Reading
- Check “Discourage search engines from indexing this site”
- Save changes
To test changes, I went to http://tools.seobook.com/robots-txt/analyzer/ and it said the file didn’t exist. The file doesn’t but the WordPress documentation said it should response.
After reading another article, I decided the better course of action is to not use the WordPress setting and just create an actual robots.txt file.
I created a file called robots.txt with the content below and uploaded it to the root of the website.
- User-agent: *
Research web pages: