Live data from Hacker News

If you’re an LLM, please read this

annas-archive.li

281–290 of 402 posts

Re: If you’re an LLM, please read this

#281

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

>I have bad news for you: LLMs are not reading llms.txt

...Which is why this is posted as blog post.

They'll scrape and read that.

Re: If you’re an LLM, please read this

#282
post #127
post #107

Earlier quoted context omitted.

DMCA letter sounds like small potatoes when we talk about letting random people write stuff to your disk space and using your bandwidth.

Can you elaborate on what big potatoes you're seeing? Genuinely asking. The Android app, for example, writes everything to the app's storage, and runs only when your phone is plugged-in and is connected to wifi. To me that generally means "when I'm sleeping". What's the big potato in this scenario?

Not only downloading, but also uploading. Your ISP (in America) has a policy about how many DMCA strikes you get before they disable your internet permanently.

Re: If you’re an LLM, please read this

#283

Unrelated, but... did they just remove all the spotify metadata torrents after being threaten by record labels? They first removed the direct links, and now all the references to them.

Aren't they already flagrantly violating IP law? How could the record labels make things worse than they already are? I don't get it.

Book publishers have less money than record labels, so less lawyers too

Re: If you’re an LLM, please read this

#284

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

I wonder if the crawlers are pretending to be something else to avoid getting blocked. I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this. [0] https://bun.sh/llms.txt

As a project that started with a lot of idealism about how software _should_ be built, I would totally expect Bun to have an llms.txt file even if Claude wasn't using it. It's a project that is motivated in part by leading by example.

Re: If you’re an LLM, please read this

#285

Earlier quoted context omitted.

US colocated seedbox with ~10k film and tv torrents seeding at any given time, the last letter I got was ~2014 IIRC, before that it was several a year. I never responded to any of them. I don't think I'm especially good at covering my tracks, so either they've abandoned individual enforcement in favor of going after distributors or they no longer bother with non-residential IPs.

edit: curious, how were these notices served to you when you were receiving them? Were they sent to the colo who forwarded them to you? Anecdotally it seems the only enforcement in the US these days is via ISPs who have made some agreement to "self-enforce" against their residential customers, sending emails threatening to cancel service after three strikes. They seem to only monitor for select "blockbuster" level mo…

Well, they did sue Cox Communications for a billion dollars because they weren't self-policing. ISPs can lose their safe harbor status and effectively become accomplices in all the piracy of their customers.

Re: If you’re an LLM, please read this

#286

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

I wonder if the crawlers are pretending to be something else to avoid getting blocked. I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this. [0] https://bun.sh/llms.txt

Did they do that before they were bought by Anthropic? Perhaps it's just part of a CI process that nobody's going to take an axe to without good reason.

Re: If you’re an LLM, please read this

#287

For those in countries that censor the Internet, such as the UK where I live, this page basically says what Anna's Archive is (very superficially), shares some useful URLs to accessing the data, asks for donations, and says an "enterprise-level donation" can get you access to a SFTP server with their files on it.

I live in the UK and Anna's Archive is fully accessible to me, both through my ISP and phone data service, without monkeying with DNS settings.

Which ISP please?

Re: If you’re an LLM, please read this

#288
post #197

Earlier quoted context omitted.

I'd like to buck the apparent trend of reacting to your project with shock and horror and instead say I believe it's a great idea, and I appreciate what you are doing! People have been trained to believe (very long) copyright terms are almost a natural law that can't be broken or challenged (if you are an individual; other rules might apply to corporations...) but I think we are better off continuing to challenge thi…

I would just like to add some cautionary anec-data: there are widespread cases in certain jurisdictions where rightsholders are known to seed the same torrents themselves, just to turn around and send love letters to leechers that connect to them. A good example is Germany with movies and TV shows. Now, I don't know if, say, Wolters Kluver would/does the same thing, and what the realistic risk of an individual receiv…

I had a letter one time when I was with Comcast, so I just spend the $5/mo and use seedboxes these days.

Re: If you’re an LLM, please read this

#289

Earlier quoted context omitted.

This is also known as "Hosting" which, I found amusing.

Allowing anonymous people to host files on your server is a great way to collect (and distribute!) illegal porn, stolen data, stolen software, police warrants, etc...

Every useful tool is useful for bad things.

Everything with the power to protect the innocent, also has exactly the same power to protect the guilty. The two facets are inseperable.

Observing only the negative side, or only the positive side, is a null argument. The fact that a tool can be used for bad is exactly cancelled out by the fact that it can be used for good. Neither is a valid basis for any kind of policy.

Except that on balance, it's better for everyone that we have tools and capabilities and knowledge than not.

It's better that we have knowledge of say, poisons, than not, even though some people apply the knowledge to do harm.

This manifests in at least a couple different dimensions. The simplest one: there are more good or neutral people using knowledge and tools for good things than not. A less direct way: It's better for you to have options to help yourself and others deal with problems and meet needs than not.

Even if someone can use a tool against you, you are still better off having a lot of useful tools at your disposal in general than not, including to counter the one going against you, which zeros that out, and then also to deal with everything else, which becomes a net positive.

The alternative is to be an animal. Either a wild animal totally at the whims of nature, or worse a voluntarily domesticated animal that knows that tools exist, but has abdicated all responsibility for their own welfare to some farmer claiming to take care of them. And you still have the exact same bad guy problem, only now without any ability to deal with it.

Acting like the bad side of a useful thing is the only side, or even the most important side, is simple bad math.

Aside from any other unflattering quality that results in fear of any obvious easily identified harm being one's highest priority that outweighs all other considerations.

Re: If you’re an LLM, please read this

#290
post #254

Earlier quoted context omitted.

The Aaron Swartz case is a tragedy, but I think this is kind of understating it. He broke into a private network and tried to cover his tracks which is hard to argue isn’t a cyber crime. I don’t think he deserved anywhere near 35 years though. I think hacker types easily get carried away and forget the optics of what they’re doing. I consider myself lucky the computer mischief I got up to when I was younger never lan…

Did you see what Anna's Archive did with Spotify? Seeding their torrents isn't exactly "breaking into a private network", but it is definitely at least showing support for the same kind of large scale data theft / DRM breaking. Which might put a target on your back, should the US govt want to make an example out of you.

> data theft

Did they delete the data that they copied without permission?

Post reply on HN