Commite0a5434

committed

ENH: add sitemap to robots.txt

1 parent1051ec9 commite0a5434Copy full SHA for e0a5434

File tree

+19

-5

lines changed

+19

-5

lines changed

Lines changed: 19 additions & 5 deletions

Original file line number	Diff line number	Diff line change
`@@ -1,13 +1,27 @@`
	`1`	`+# Docs: https://developers.google.com/search/docs/advanced/robots/intro`
	`2`	`+# Note old files will still be indexed if they have links to them,`
	`3`	`+# hopefully they are weighted less...`
	`4`	`+`
`1`	`5`	`User-agent:*`
	`6`	`+`
	`7`	`+# do not search root directory by default.`
`2`	`8`	`Disallow:/`
`3`		`-Disallow:/examples/ # outdated location. Current examples are in /gallery/.`
`4`		`-Allow:/cmocean/`
`5`		`-Allow:/mpl-probscale/`
`6`		`-Allow:/mpl_toolkits/`
`7`		`-Allow:/xkcd/`
	`9`	`+`
	`10`	`+# but allow some subdirectories:`
`8`	`11`	`Allow:/stable/`
	`12`	`+`
	`13`	`+# versions should be searched. Note sitemap.xml downweights them...`
`9`	`14`	`Allow:/5.*/`
`10`	`15`	`Allow:/4.*/`
`11`	`16`	`Allow:/3.*/`
`12`	`17`	`Allow:/2.*/`
`13`	`18`	`Allow:/1.*/`
	`19`	`+`
	`20`	`+# packages that should still be searched:`
	`21`	`+Allow:/cmocean/`
	`22`	`+Allow:/mpl-probscale/`
	`23`	`+Allow:/mpl_toolkits/`
	`24`	`+Allow:/xkcd/`
	`25`	`+`
	`26`	`+# tell robots this is sitemap`
	`27`	`+Sitemap:https://matplotlib.org/sitemap.xml`

Comments

(0)