[Gastongr]

The proper usage of the robots.txt file

Here is a list of variables that you can include in a robots.txt file and there meaning:


1)User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.

2)Disallow: In the field you specify the files and folders not to include in the crawl.

3)The # is to represent comments



Here are some examples of a robots.txt file



User-agent: *

Disallow:



The above would let all spiders index all content.



Here another



User-agent: *

Disallow: /cgi-bin/



The above would block all spiders from indexing the cgi-bin directory.



User-agent: googlebot

Disallow:



User-agent: *

Disallow: /admin.php

Disallow: /cgi-bin/

Disallow: /admin/

Disallow: /stats/



In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.





Jimmy Whisenhunt is the webmaster at http://www.vipenterprises.org VIP Enterprises