|
|
#1 |
![]() |
The proper usage of the robots.txt file
Here is a list of variables that you can include in a robots.txt file and there meaning: 1)User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example. 2)Disallow: In the field you specify the files and folders not to include in the crawl. 3)The # is to represent comments Here are some examples of a robots.txt file User-agent: * Disallow: The above would let all spiders index all content. Here another User-agent: * Disallow: /cgi-bin/ The above would block all spiders from indexing the cgi-bin directory. User-agent: googlebot Disallow: User-agent: * Disallow: /admin.php Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /stats/ In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php. Jimmy Whisenhunt is the webmaster at http://www.vipenterprises.org VIP Enterprises |
|
|
|
![]() |
| Tags |
| file, proper, robotstxt, usage |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| File upload help | tbobker | Programming Forum | 1 | 01-30-2010 06:47 AM |
| Converting an .avi file to .swf? | duran | Graphic Design Forum | 2 | 10-26-2009 07:52 AM |
| Including header file using HTML | Wladna | HTML, DHTML, CSS And JavaScript | 1 | 12-05-2006 06:42 AM |
| Delete Contents From A *.txt File. | Samm | Programming Forum | 2 | 07-24-2006 06:40 PM |
| Free File HOsting | extreme90 | Offers And Request | 0 | 12-31-2005 08:55 AM |









