Understanding the robots.txt file from joed.com
One item most webmasters forget is the robots.txt file. This is a file that almost every search engine spider looks at to see how to index your website. This file lets you tell the spiders how to index your site. To see if it exists on your website just go to your main site. http://www.joed.com/ and then add robots.txt to the end of your url like this: http://www.joed.com/robots.txt. If you don’t find it on your site I will guarentee that you are losing business. I have seen spiders leave a site because it doesn’t exist and not come back again for months.
To create a file for your site just follow these simple steps:
1. Create a txt file in notepad called robots.txt
2. Mine are similar to this. Note the crawl delay. On smaller servers running a couple hundred sites if the spiders are hitting the sites hard they can actually crash a server, so on some of my sites I use the delay below.
User-agent: *
Disallow: /cgi-bin/
Disallow: /cgibin/
User-agent: msnbot
Crawl-delay: 10
User-agent: Teoma
Crawl-delay: 10
User-agent: Slurp
Crawl-delay: 10
3. Then load that file into your main web directory.
For more info on the robots.txt standards and options you can take a look at: http://www.robotstxt.org/
If you are running your own servers you can also set up just one robots.txt file that will work for ALL your sites.
Copy the file into your /home/sites/home/web/ directory. On some boxes web could be replaced with html_docs or www or whatever your provider uses as the servers root dir. This isnt something to mess with if you arent sure of what you are doing.
Add the Aliases for robots.txt in either the httpd.conf file or the srm.conf file. It should look something like this:
Alias /robots.txt “/home/sites/home/web/robots.txt”
Do httpd -t before you restart This will check for any errors in your apache config files.
Restart your services or reboot your box. If everything is where it is supossed to be then your robots.txt file should work for every domain on your server.
NOTE: HAVING A ROBOTS META TAG DOES NOT REPLACE THE ROBOTS.TXT FILE!
Please check out my My Ebay Store for Search Engine Optimization . for more information.