Webmaster Forum > Website Design and Graphics > Search Engine Optimization >

The proper usage of the robots.txt file


Reply
 
Share Thread Thread Tools
Old 04-26-2005, 08:36 PM   #1
admin

 
admin's Avatar
 
Join Date: Apr 2004
Location: La Plata
Age: 23
Posts: 1,055
Reputation: admin has a spectacular aura aboutadmin has a spectacular aura about
iTrader: (0)
Default The proper usage of the robots.txt file

The proper usage of the robots.txt file

Here is a list of variables that you can include in a robots.txt file and there meaning:


1)User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.

2)Disallow: In the field you specify the files and folders not to include in the crawl.

3)The # is to represent comments



Here are some examples of a robots.txt file



User-agent: *

Disallow:



The above would let all spiders index all content.



Here another



User-agent: *

Disallow: /cgi-bin/



The above would block all spiders from indexing the cgi-bin directory.



User-agent: googlebot

Disallow:



User-agent: *

Disallow: /admin.php

Disallow: /cgi-bin/

Disallow: /admin/

Disallow: /stats/



In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.





Jimmy Whisenhunt is the webmaster at http://www.vipenterprises.org VIP Enterprises
admin is offline   Reply With Quote
Old 04-13-2010, 09:59 AM   #2
salenemendes

Junior Member
 
Join Date: Apr 2010
Location: Alamo
Age: 30
Posts: 11
Reputation: salenemendes is on a distinguished road
iTrader: (0)
Default Re: The proper usage of the robots.txt file

Robots.txt file usage is sometimes ignored. On the other hand, it is an important factor for the webpages being indexed properly and very easy to setup.

I know that robots.txt is not something new. But, I’ve been preparing a SEO sheet for a while and wanted to share this small & useful portion with you. Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.

In general, we prefer that our webpages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Like the personal images folder, website administration folder, customer’s test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don’t want them to be indexed.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
salenemendes is offline   Reply With Quote
Old 04-13-2010, 11:51 PM   #3
hellokitty17

 
Join Date: Jan 2010
Location: Orange
Posts: 173
Reputation: hellokitty17 is on a distinguished road
iTrader: (0)
Default Re: The proper usage of the robots.txt file

Thats a good point there.. These Robot.txt files in my understanding helps hide those files you dont wanna be indexed.? Am i right.? Well if that so.. Its good and maybe id be using it too.!
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
hellokitty17 is offline   Reply With Quote
Old 04-15-2010, 12:56 PM   #4
andrewgunn4u

Junior Member
 
Join Date: Apr 2010
Location: alabama
Age: 30
Posts: 7
Reputation: andrewgunn4u is on a distinguished road
iTrader: (0)
Default Re: The proper usage of the robots.txt file

yes robots place a very important role in search engine crawling.
andrewgunn4u is offline   Reply With Quote
Old 04-20-2010, 09:59 AM   #5
soromaru77

Junior Member
 
Join Date: Apr 2010
Location: Semarang
Age: 33
Posts: 3
Reputation: soromaru77 is on a distinguished road
iTrader: (0)
Default Re: The proper usage of the robots.txt file

Robots.txt indeed play a a bit of role in SEO.
soromaru77 is offline   Reply With Quote
Old 04-21-2010, 02:14 AM   #6
heebdawq16

 
heebdawq16's Avatar
 
Join Date: Jul 2009
Location: florida
Posts: 138
Reputation: heebdawq16 is on a distinguished road
iTrader: (0)
Default Re: The proper usage of the robots.txt file

The robots.txt file are primarily used to indicate to spiders index certain of your pages.

But I found that spiders usually pay more attention to the robots meta tags.

Yahoo should be following robots.txt rules.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
& Transportation info
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
& Travel Guides
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
heebdawq16 is offline   Reply With Quote
Reply

Tags
file, proper, robotstxt, usage


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
File upload help tbobker Coding Forum 3 04-05-2010 10:37 PM
Use this script to call robots to your site... ovaismirza22 Search Engines Forum 1 02-11-2010 02:40 PM
PHP code to upload file Tharaka Deshan PHP Forum 0 01-20-2010 01:59 AM
Converting an .avi file to .swf? duran Graphics Forum 2 10-26-2009 06:52 AM
Delete Contents From A *.txt File. Samm Coding Forum 2 07-24-2006 05:40 PM

Show/Hide Menu
Webmaster Help

You are currently viewing our webmaster forum as a guest, with limited access to our forums and webmaster resources.

As a registered member of our webmaster forums you gain useful webmaster help and the ability to participate in friendly informative webmaster discussion with webmasters across the web.

If you have any problems with registration or your account login, please contact support.
Log in

Set Cookie?


Sites We Like

Ads

Copyright © 2004 - 2010 Webmaster Help. All Rights Reserved.
Exdevio Webmaster Network