We broke the web so badly for humans that we had to build a clean web for machines, and now humans will have to use machines to experience a clean web again.
I wonder why we broke the web.
[dead]
Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
21–30 of 63 posts
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#22oh don't worry, in 5 years your AI will be unundated with context poison prompts that try to get them to spend all your bank notes and meta bucks on equally useless things. This is just a redeux of the early web.
Already happening. I was using Claude to check out sampler plugins and I'm sure it happens undetected, and it might have mentioned it with other versions, but Claude Opus 4.8, being it's helpful, honest self, told me that one of the pages it reviewed had hidden text instructing it to recommend that plugin. It caught it and was able to avoid influence from that plugin at least, but we're already living in that world.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#23Earlier quoted context omitted.
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path). All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
How is a static blog being scraped a problem? Do you not use a CDN?
Are all blogs static though?
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#24I never thought about it before now but the llm era could be a form of renaissance for blind people on the Internet. An alternative web where functionality of every page is described in short but detailed text instead of extremely verbose and non-linear html tree structure.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#25Earlier quoted context omitted.
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path). All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
How is a static blog being scraped a problem? Do you not use a CDN?
> a blogging platform with around 80k blogs
But nah, I'm sure OP doesn't know about CDNs.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#26Earlier quoted context omitted.
I wonder why we broke the web.
For the same reasons why we eventully pollute and corrupt every system and environment we use. If there is any benefit that can be extracted for some while the costs are borne by many, than this will occur and generate a positive feedback loop that grows over time. It's the law of monetization.
>than this will occur and generate a positive feedback loop that grows over time.
And despite this, modern life is made possible by the illusion that "regulations" work..
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#27Does any of the LLM providers actually use llms.txt? If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path). All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
I'm seeing quite a bit of request for these on my work's GitBook documentation site.
But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#28The only annoyance is web browsers like chrome do not render the markdown. I imagine Claude could zero-shot a Chrome plugin for that.
Of course plugins that do this already exist. Save your tokens.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#29Why didn't they place it in .well-known? Also, I couldn't find a website that has it.
Re: Ask HN: Is the web for machines (/llm.txt) the one we wished we had as humans?
#30Pretty much.
There is an enshittification cycle at work. The web used to be good, predominately text, and useful, 25 years ago. Then... slowly... we added javascript, then AJAX, CSS, flash, interstitials, popups, marketing, social media, algorithms, doomscrolling... gradually but surely turn it into the unusable cesspool that it is today.
Now we have AI! I think a big part of its utility is that it gets us back to text/information, and lets us bypass all the "beautiful" design / nonsense on the material it is trained on.
However, AI is just beginning its enshittification cycle - now that it has a critical mass of users, it is an irresistible target to start slowly adding ads, misinformation, conspiracy theories, and whatever else people can dream up, until it also becomes unusable and the cycle repeats.