This is thetalk page for discussing improvements to theRobots.txt article. This isnot a forum for general discussion of the article's subject.

Put new text under old text.Click here to start a new topic. New to Wikipedia? Welcome!Learn to edit;get help. Assume good faith Be polite andavoid personal attacks Be welcoming to newcomers Seekdispute resolution if needed Article policies Neutral point of view No original research Verifiability

Find sources: Google (books ·news ·scholar ·free images ·WP refs) ·FENS ·JSTOR ·TWL

Archives:1Auto-archiving period:6 months

This article is ratedC-class on Wikipedia'scontent assessment scale. It is of interest to the followingWikiProjects:

ComputingMid‑importance This article is within the scope ofWikiProject Computing, a collaborative effort to improve the coverage ofcomputers,computing, andinformation technology on Wikipedia. If you would like to participate, please visit the project page, where you can jointhe discussion and see a list of open tasks.ComputingWikipedia:WikiProject ComputingTemplate:WikiProject ComputingComputing Mid This article has been rated asMid-importance on theproject's importance scale. InternetHigh‑importance This article is within the scope ofWikiProject Internet, a collaborative effort to improve the coverage of theInternet on Wikipedia. If you would like to participate, please visit the project page, where you can jointhe discussion and see a list of open tasks.InternetWikipedia:WikiProject InternetTemplate:WikiProject InternetInternet High This article has been rated asHigh-importance on theproject's importance scale.

On 18 December 2022, it was proposed that this article bemoved fromRobots exclusion standard torobots.txt. The result ofthe discussion wasmoved.

Hello,

I would like to propose the inclusion of a free tool as an additional resource on the robots.txt Wikipedia page. The tool, available atgeneraterobotstxt.com, helps users easily create and optimize their robots.txt files.

The purpose of this tool aligns with the educational and practical goals of the Wikipedia page. It allows users to:

• Automatically generate valid robots.txt files tailored to their needs.
• Avoid common errors that may inadvertently block or allow undesired web and AI crawlers.
• Learn about the purpose and structure of this file through a user-friendly interface.

This tool is freely accessible and designed to benefit individuals with varying levels of technical expertise. It has received positive feedback for its simplicity and usefulness.

I believe including this resource would complement the information provided in the article and support users looking for practical guidance.

Looking forward to the community's thoughts on this proposal.

Thank you!83.54.17.114 (talk)21:54, 19 December 2024 (UTC)[reply]

I would say no because of theWP:ELNO policy. And also because it would be unfair to other tool developers.PhotographyEdits (talk)19:48, 20 December 2024 (UTC)[reply]

• C-Class Computing articles
• Mid-importance Computing articles
• All Computing articles
• C-Class Internet articles
• High-importance Internet articles
• WikiProject Internet articles