Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

91–100 of 191 posts

Re: Llms.txt

#91

Post by Tepix »

JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

I think the whole idea of extra instructions required for LLMs is unnecessary. A decent LLM should be able to handle browsing the site, if needed it can use the sitemap. It can hopefully also figure out what the various sections are about.

Re: Llms.txt

#92

Post by Tepix »

otherme123 wrote: post #63
jraph wrote: post #61

Earlier quoted context omitted.

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing. if ($http_user_agent ~ facebook) { return 444; } if ($http_user_agent ~ Amazonbot) { return 444…

In your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { default 0; ~facebook 1; [...] ~PerplexityBot 1; } In your site.conf, server block, add if ($badagent) { return 444; }

Ideally i would like these crawlers to access /robots.txt but nothing else.

Only if they ignore robots.txt the access rules will stop them.

Re: Llms.txt

#93

Post by KaiserPro »

Its a nice idea, but ultimately pointless.

OpenAI have admitted that they are routinely breaking copyright licenses, and not very many people are taking them to court to stop. Its the same for most other LLM trainers who don't have thier own content to use (ie anyone other than meta and google)

Unless a big company takes umbridge, then they will continue to rip content.

THe reason they can get away with it is that unlike with napster in the late 90s, the entertainment industry can see a way to make money off AI generated shite. So they are willing to let it slide in the hopes that they can automate a large portion of content creation.

Re: Llms.txt

#94

Post by yochem »

jph00 wrote: post #85
romantomjak wrote: post #78

I find it confusing that author proposes llms.txt, but the content is actually markdown? I get that they tried to follow the convention, but then why not make it a simple text file like the robots.txt is?

Markdown is plain text. llms.txt is meant to be displayed in plain text format, not rendered to html.

Yes, and .py is "plain" text too. The extension however helps with signaling the intend of the file. Also, there is something to say for the argument "there is no such thing as plain text" [0]

[0]: https://youtu.be/gd5uJ7Nlvvo

Re: Llms.txt

#95

Post by rixrax »

Rolls sleeves up to start working on custom GPT and training my own LLM to offer service to produce llms.txt for a website by letting them process the website... ;-)

Re: Llms.txt

#96

Post by gitgud »

itsbjoern wrote: post #49
internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

Always find it amusing when people write about „blocking“ requests using robots.txt as if they are deploying a firewall

Agreed, all it takes is another site to copy the content, then an LLM could just scrape that…

An open web that block scraping… is likely “not an open web”

Re: Llms.txt

#98

Post by arnaudsm »

I love minimalistic specs like this. I miss the 90s lightweight internet, that projects like gopher and Gemini try to resurrect.

But it's going against 2 trends :

- Every site needs to track and fingerprint you to death with JS bloatware for $

- LLMs break the social contract of the internet: hyperlinking is a two way exchange, LLM RAG is not. No attribution, no ads, basically theft. Walled gardens will never let this happen. And even a hobbyist like myself doesn't want to

Re: Llms.txt

#99

Post by msp26 »

jph00 wrote: post #82
mrweasel wrote: post #79

Wouldn't this open up for manipulating LLMs? You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares no resemblance to the actual content of the page. If you really care about your content being picked up by the scrapers, why not structure it better? Most of the LLMs are pretty much black boxes, so we don't really know what a better structure would look like, b…

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.

> presumably if you create an llms.txt you're not looking to manipulate anyone

I don't think the public sentiment around scrapers and LLMs is that friendly.

I personally think scraping is very important and it allows for smaller players to compete (as long as you're careful about the number of requests).

Re: Llms.txt

#100

Post by idf00 »

mrweasel wrote: post #87
jph00 wrote: post #82

Earlier quoted context omitted.

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.

> if you create an llms.txt you're not looking to manipulate anyone You don't know me :-) My suggestion is that someone might want taint the data that goes into an LLM. Let's say you have a website with guides, examples and tips and tricks for writing bash. What would prevent you from pointing the LLMs to separate content which would contain broken examples and code with a number of security issues, because you long…

llms.txt doesn't seem to make it any easier or harder to do that.
Post reply on HN