Commitf733e32

committed

Add robots.txt for testing purposes

1 parentc02222e commitf733e32Copy full SHA for f733e32

File tree

+10

-0

lines changed

+10

-0

lines changed

Lines changed: 10 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,10 @@`
	`1`	`+# Used by Lib/test/test_robotparser.py`
	`2`	`+`
	`3`	`+# No one should be crawling us with Nutch.`
	`4`	`+User-agent:Nutch`
	`5`	`+Disallow:/`
	`6`	`+`
	`7`	`+# Hide old versions of the documentation and various large sets of files.`
	`8`	`+User-agent:*`
	`9`	`+Disallow:/~guido/orlijn/`
	`10`	`+Disallow:/webstats/`

Comments

(0)