Llms.txt

1. Creating a tool based on helping llms train on website implies that: llms have a problem with training on websites (even though html is designed for easy machine parsing of content) and second that llms are still crawling and have not moved on to other harder sources of data.

2. I am challenging those raison d'etre assumptions on the tool. Questioning not only the tool and its usefulness, but its creator's understanding of the state of llm development.

Llms do not use an alternative llm only channel or api for communication.

Llms have no problem reading a website, if you implement this dumb standard as a website owner you are duplicating the things an llm can read.

And if you implement this as an llm tool dev, you are now reading two different sources of information, you are now tasked with integrating them and resolving differences, and opening yourself up to straight up lying.

If a website says one thing to humans and another to llms, which one would you rather display to the user? That's right, the thing humans actually see.

If llms benefit from a standarized side channel for transmitting metadata, it needs to:

1-not be the actual data 2- be a bit more explicit about what data is transmitted. This standard proposes syntax but leaves actual keys up to the user? Sections are called Optional, docs, FastHTML?

Have some balls pick specific keys and bake them into your proposal, and be specifically useful. Sections like: copyright policy, privacy policy, sourcing policy, crowdsourcing, legal jurisdiction, owner. Might all be useful, although they would not strictly be llm only.

What are you talking about? The tool has basically nothing to do with websites, other than it is assumed the author of the document will provide it to the user via their website and that the user will know to find it there. Technically speaking, the user could, instead, request the document from the author over email, fax, or even a letter delivered by hand. But HTTP is more convenient for a number of reasons.

> llms have a problem with training on websites

If you mean LLMs have a problem with keeping up with current events, yes, that is essentially the problem this is intended to solve. It offers a document you can inject into your prompt (think RAG) that provides current information that an LLM is probably not up-to-date with – that it can use to gain knowledge about information that may not have even existed a minute ago.

You could go to the regular HTML website and copy/paste the content out of page after page after page to much the same effect, but consolidating it all into one place, with an added bonus of being without any extraneous information that might eat up tokens, to copy/paste once makes it easier for the user.

> Questioning not only the tool and its usefulness

Its usefulness is worth questioning. It very well may not be useful, and the author who proposed this even admits it may not be useful – putting it out there merely to test the waters to see if anyone finds it to be. But your questions are a long way away from being relevant to the tool and how it might potentially be useful.

Some humans do for websites; I personally couldn't care less and I find it often just annoying / in the way. I wish all sites where just black on white or the reverse and with clear interaction elements (including for saas sites). I welcome the near future where I can say; 'show me all important sentry issues, ah yes, make an issue in github to to fix this one and just make the rest resolved' instead of having to click through a myriad of useless and often confusing 'UX' and 'beauty' just to do things.

Non saas sites I just visit to read so I immediately ram the reader-mode button which shows the site indeed as I want.