Robots txt

[lostinfr]

I'm a bit stuck on the robots.txt :confused:
If I wanted to stop spiders from visiting:
http:/ww.me.com/modules/pbboard/posting.php would I put: Disallow: /modules/pbboard/posting.php or would that stop the spider from crawling anything in that directory?

[rds]

Hi

The disallow field prevents a spider from visiting the named url.
For example
disallow /stats will disallow any thing beginning with /stats
such as /stats.htm and /stats/*

whereas
disallow /stats/ will allow /stats.htm but disallow anything in
/stats/ susch as /stats/index.html

You can also find more help here
http://www.robotstxt.org/wc/norobots.html

Hope that helps.

Andre
http://www.rds.co.za

[lostinfr]

Thanks :) :)

[lostinfr]

Hi,

I think I've done it right :confused: or will this stop google from spidering my entire site? Also if I just wanted to stop it spidering any php pages what would I put in? (Some of the sites got a mod-rewrite hence both php & html)

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/
Disallow: /forum/
Disallow: /Test/
Disallow: /zoom_search/
Disallow: /templates/
Disallow: /entropybanner/
Disallow: /flash_games/
Disallow: /random/
Disallow: /spelling/
Disallow: /modules/pbboard/search.php
Disallow: /modules/pbboard/search.htm
Disallow: /modules/pbboard/images
Disallow: /modules/pbboard/language
Disallow: /modules/pbboard/viewonline.php
Disallow: /modules/pbboard/login.php
Disallow: /login.php
Disallow: /login.php
Disallow: /user.html
Disallow: /register.html
Disallow: /user.php
Disallow: /search.html
Disallow: /modules/pbboard/memberlist.php
Disallow: /amevents+submit.htm
Disallow: /search.php
Disallow: /userinfo/
Disallow: /modules/pbboard/privmsg.php
Disallow: /modules/pbboard/privmsg.htm
Disallow: /modules/pbboard/faq.php