[admin]

The proper usage of the robots.txt file

Here is a list of variables that you can include in a robots.txt file and there meaning:


1)User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.

2)Disallow: In the field you specify the files and folders not to include in the crawl.

3)The # is to represent comments



Here are some examples of a robots.txt file



User-agent: *

Disallow:



The above would let all spiders index all content.



Here another



User-agent: *

Disallow: /cgi-bin/



The above would block all spiders from indexing the cgi-bin directory.



User-agent: googlebot

Disallow:



User-agent: *

Disallow: /admin.php

Disallow: /cgi-bin/

Disallow: /admin/

Disallow: /stats/



In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.





Jimmy Whisenhunt is the webmaster at http://www.vipenterprises.org VIP Enterprises

[salenemendes]

Robots.txt file usage is sometimes ignored. On the other hand, it is an important factor for the webpages being indexed properly and very easy to setup.

I know that robots.txt is not something new. But, I’ve been preparing a SEO sheet for a while and wanted to share this small & useful portion with you. Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.

In general, we prefer that our webpages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Like the personal images folder, website administration folder, customer’s test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don’t want them to be indexed.

[hellokitty17]

Thats a good point there.. These Robot.txt files in my understanding helps hide those files you dont wanna be indexed.? Am i right.? Well if that so.. Its good and maybe id be using it too.!

[andrewgunn4u]

yes robots place a very important role in search engine crawling.

[soromaru77]

Robots.txt indeed play a a bit of role in SEO.

[heebdawq16]

The robots.txt file are primarily used to indicate to spiders index certain of your pages.

But I found that spiders usually pay more attention to the robots meta tags.

Yahoo should be following robots.txt rules.