Robots.txt Files Can Make Your Life Easier

According to Keith Hogan from Ask:

i) Less than 35% of websites have a robots.txt file
ii) The majority of robots.txt files are copied from others found online
iii) On many occasions robots.txt files are provided by your web hosting service

If your website or blog uses a lot of images that you don’t want crawled or indexed by the search engines’ photo search features, you can build a page of photos and include a robots.txt file to tell the search engines not to crawl that page. This is extremely important. It can be used for any content that is for your personal use but you don’t want to the world to see.

Add a page to your website but don’t link it to any other page on the website. It’s basically a file on your domain. Then, put a robots.txt file on that page to notify search engines that you don’t want it crawled. In this manner you can keep photos, video files, podcasts, newsletters, links, anything that you want protected and not indexed without keeping it all on your hard drive. This is a very effective strategy for bloggers and others who use material on a regular basis but need a way to access it easily and quickly.

Leave a Reply

Your email address will not be published.