llms.txt is designed to coexist with current web standards. While sitemaps list all pages for search engines,`llms.txt` offers a curated overview for LLMs. It can complement robots.txt by providing context for allowed content. The file can also reference structured data markup used on the site, helping LLMs understand how to interpret this information in context.

The approach of standardising on a path for the file follows the approach of`/robots.txt` and`/sitemap.xml`. robots.txt and`llms.txt` have different purposes---robots.txt is generally used to let automated tools what access to a site is considered acceptable, such as for search indexing bots. On the other hand,`llms.txt` information will often be used on demand when a user explicitlyrequesting information about a topic, such as when including a coding library's documentation in a project, or when asking a chat bot with searchfunctiontionality for information. Our expectation is that`llms.txt` will mainly be useful for*inference*, i.e. at the time a user is seeking assistance, as opposed to for*training*. However, perhaps if`llms.txt` usage becomes widespread, future training runs could take advantage of the information in`llms.txt` files too.

The approach of standardising on a path for the file follows the approach of`/robots.txt` and`/sitemap.xml`. robots.txt and`llms.txt` have different purposes---robots.txt is generally used to let automated toolsknowwhat access to a site is considered acceptable, such as for search indexing bots. On the other hand,`llms.txt` information will often be used on demand when a user explicitlyrequests information about a topic, such as when including a coding library's documentation in a project, or when asking a chat bot with searchfunctionality for information. Our expectation is that`llms.txt` will mainly be useful for*inference*, i.e. at the time a user is seeking assistance, as opposed to for*training*. However, perhaps if`llms.txt` usage becomes widespread, future training runs could take advantage of the information in`llms.txt` files too.

sitemap.xml is a list of all the indexable human-readable information available on a site. This isn’t a substitute for`llms.txt` since it:

- Often won’t have the LLM-readable versions of pages listed

- Doesn’t include URLs to external sites, evenalthough they might be helpful to understand the information

- Doesn’t include URLs to external sites, eventhough they might be helpful to understand the information

- Will generally cover documents that in aggregate will be too large to fit in an LLM context window, and will include a lot of information that isn’t necessary to understand the site.

- [Starlette full documentation](https://gist.githubusercontent.com/path/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development.

To create effective`llms.txt` files, consider these guidelines: Use concise, clear language. When linking to resources, include brief, informative descriptions. Avoid ambiguous terms or unexplained jargon. Run a tool that expands your`llms.txt` file into an LLM context file and test a number of language models to see if they can answer questions about your content.

To create effective`llms.txt` files, consider these guidelines:

- Use concise, clear language.

- When linking to resources, include brief, informative descriptions.

- Avoid ambiguous terms or unexplained jargon.

- Run a tool that expands your`llms.txt` file into an LLM context file and test a number of language models to see if they can answer questions about your content.