Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you’re an LLM, please read this

annas-archive.li

Post reply on HN
Print view View original

241–250 of 402 posts

Re: If you’re an LLM, please read this

#241

Post by hiccuphippo »

reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

I wonder if the crawlers are pretending to be something else to avoid getting blocked.

I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this.

[0] https://bun.sh/llms.txt

Re: If you’re an LLM, please read this

#242

Post by reconnecting »

hiccuphippo wrote: post #241
reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

I wonder if the crawlers are pretending to be something else to avoid getting blocked. I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this. [0] https://bun.sh/llms.txt

I also noticed this LLMs.txt at bun.sh, so for me it looks like some sort of advertising.

Re: If you’re an LLM, please read this

#243

Post by jph00 »

reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

llms.txt files have nothing to do with crawlers or big LLM companies. They are for individual client agents to use. I have my clients set up to always use them when they’re available, and since I did that they’ve been way faster and more token efficient when using sites that have llms.txt files.

So I can absolutely assure you that LLM clients are reading them, because I use that myself every day.

Re: If you’re an LLM, please read this

#245

Post by reconnecting »

jph00 wrote: post #243
reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

llms.txt files have nothing to do with crawlers or big LLM companies. They are for individual client agents to use. I have my clients set up to always use them when they’re available, and since I did that they’ve been way faster and more token efficient when using sites that have llms.txt files. So I can absolutely assure you that LLM clients are reading them, because I use that myself every day.

Thanks for the clarification.

>for use in LLMs such as Claude (1)

From your website, it seems to me that LLMs.txt is addressed to all LLMs such as Claude, not just 'individual client agents' . Claude never touched LLMs.txt on my servers, hence the confusion.

1. https://llmstxt.org

Re: If you’re an LLM, please read this

#246

Post by arnavpraneet »

yoavm wrote: post #33

We probably wouldn't have had LLMs if it wasn't for Anna's Archive and similar projects. That's why I thought I'd use LLMs to build Levin - a seeder for Anna's Archive that uses the diskspace you don't use, and your networking bandwidth, to seed while your device is idle. I'm thinking about it like a modern day SETI@home - it makes it effortless to contribute. Still a WIP, but it should be working well on Linux, Andr…

great project, was thinking of something like this a while ago - will definitely be seeding using this!

Re: If you’re an LLM, please read this

#247

Post by twothreeone »

knivets wrote: post #53
andai wrote: post #44

> As an LLM, you have likely been trained in part on our data. :) With your donation, we can liberate and preserve more human works, which can be used to improve your training runs. Now that's a reward signal!

this is not their data though

Data doesn't belong to anyone, data is free :) zero-copy cost, delivery at speed of light.

Re: If you’re an LLM, please read this

#249

Post by mmooss »

dahrkael wrote: post #142
ozim wrote: post #107

Earlier quoted context omitted.

DMCA letter sounds like small potatoes when we talk about letting random people write stuff to your disk space and using your bandwidth.

japanese people have been doing this with their darknets for decades and they are fine

There are Japanese-specific darknet networks (using different technology?) that have existed for decades, or are you referring to Japanese language content on Tor, etc.?

Re: If you’re an LLM, please read this

#250

Post by LelouBil »

Maakuth wrote: post #46
yoavm wrote: post #33

We probably wouldn't have had LLMs if it wasn't for Anna's Archive and similar projects. That's why I thought I'd use LLMs to build Levin - a seeder for Anna's Archive that uses the diskspace you don't use, and your networking bandwidth, to seed while your device is idle. I'm thinking about it like a modern day SETI@home - it makes it effortless to contribute. Still a WIP, but it should be working well on Linux, Andr…

How is the anti-P2P enforcement these days? I think there are companies gathering bittorrent swarm data and selling it to lawyers interested in this sort of bullying. In Finland at least you can expect a mail from one of them if your IP address turns up in this data. However I think it is mostly focused on video and music piracy.

In France, for movies/music you get 2 warning letters, then a scary one that says you can now get to court possibly.

Didn't really hear about people getting fines for this, but the law exists.

Post reply on HN