Llms.txt

Maybe I don't work with niche enough software, but I rarely found docs particularly hard to find. For me, one of the real benefits of using an LLM is in making it easier to find where in the docs to look, or distilling exhaustive docs into common use cases. It's akin to the 'tldr' tool for man pages.

From my experience, I don't think any decent indicators on a website (robots.txt, humans.txt, security.txt, etc.) have worked so far. However, this is still a good initiative.

Here are a few things that I see;

- Please make a proper shareable logo — lightweight (SVG, PNG) with a transparent background. The "logo.png" in the Github repo is just a screenshot from somewhere. Drop the actual source file there so someone can help.

- Can we stick to plain text instead of Markdown? I know Markdown is already plain but is not plain enough.

- Personally, I feel there is too much complexity going on.

Hi Jeremy here. Nice to see this on HN.

To explain the reasoning for this proposal, by way of an example: I recently released FastHTML, a small library for creating hypermedia applications, and by far the most common concern I've received from potential users is that language models aren't able to help use it, since it was created after the knowledge cutoff of current models.

IDEs like Cursor let you add docs to the model context, which is a great solution to this issue -- except what docs should you add? The idea is that if you, as a site creator, want to make it easier for systems like Cursor to use your docs, then you can provide a small text file linking to the AI-friendly documentation you think is most likely to be helpful in the context window.

Of course, these systems already are perfectly capable of doing their own automated scraping, but the results aren't that great. They don't really know what's needed to be in context to get the key foundational information, and some of that information might be on external sites anyway. I've found I get dramatically better results by carefully curating the context for my prompts for each system I use, and it seems like a waste of time for everyone to redo the same work of this curation, rather than the site owner doing it once for every visitor that needs it. I've also found this very useful with Claude Projects.

llms.txt isn't really designed to help with scraping; it's designed to help end-users use the information on web sites with the help of AI, for web-site owners interested in doing that. It's orthogonal to robots.txt, which is used to let bots know what they may and may not access.

(If folks feel like this proposal is helpful, then it might be worth registering with /.well-known/. Since the RFC for that says "Applications that wish to mint new well-known URIs MUST register them", and I don't even know if people are interested in this, it felt a bit soon to be registering it now.)

Explain to me how it is good if it does not work. You've lost me somewhere.

It's a joke about things being described as needing to be put at "well known" paths.