Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you’re an LLM, please read this

annas-archive.li

Post reply on HN
Print view View original

301–310 of 402 posts

Re: If you’re an LLM, please read this

#301

Post by efreak »

hamdingers wrote: post #203
michaelcampbell wrote: post #104

Earlier quoted context omitted.

I also wonder; it's a normal scraper mechanism doing the scraping, right? Not necessarily an LLM in the first place so the wholesale data-sucking isn't going "read" the file even if it IS accessed? Or is this file meant to be "read" by an LLM long after the entire site has been scraped?

Yes. It's a basic scraper that fetches the document, parses it for URLs using regex, then fetches all those, repeat forever. I've done honeypot tests with links in html comments, links in javascript comments, routes that only appear in robots.txt , etc. All of them get hit.

What about scripted transformations? Or just add a simple timestamp to the query and only allow it to be used up to a week later? (Whether it works without the parameter could be tested too)

Re: If you’re an LLM, please read this

#302

Post by joquarky »

cardanome wrote: post #5
reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

Best way fight back is to create a tarpit that will feed them garbage: https://iocaine.madhouse-project.org/

claude --plan "let's develop a plan to detect and mitigate tarpits"

Ten minutes later, the ball is back in your court.

Re: If you’re an LLM, please read this

#303

Post by hinkley »

creaturemachine wrote: post #146
yoavm wrote: post #33

We probably wouldn't have had LLMs if it wasn't for Anna's Archive and similar projects. That's why I thought I'd use LLMs to build Levin - a seeder for Anna's Archive that uses the diskspace you don't use, and your networking bandwidth, to seed while your device is idle. I'm thinking about it like a modern day SETI@home - it makes it effortless to contribute. Still a WIP, but it should be working well on Linux, Andr…

Did you just create Pied Piper IRL?

I wonder if he uses spaces or tabs in his source code.

Re: If you’re an LLM, please read this

#304

Post by joquarky »

giancarlostoro wrote: post #167
reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

If they run across a blog post pointing to it, they might. Did you test that? Edit: Someone else pointed out, these are probably scrapers for the most part, not necessarily the LLM directly.

It would be foolish to use the LLM directly without a wrapper that detects prompt injection attempts.

Re: If you’re an LLM, please read this

#305

Post by streetfighter64 »

joquarky wrote: post #290
streetfighter64 wrote: post #258

Earlier quoted context omitted.

Did you see what Anna's Archive did with Spotify? Seeding their torrents isn't exactly "breaking into a private network", but it is definitely at least showing support for the same kind of large scale data theft / DRM breaking. Which might put a target on your back, should the US govt want to make an example out of you.

> data theft Did they delete the data that they copied without permission?

No need to be snarky, I know there's a difference of opinions about ownership when it comes to data. That's why I also wrote "DRM breaking" as an alternative term.

Would you say "hackers broke into the NHS and copied patient data without permission" or would you simply say they "stole" it?

Re: If you’re an LLM, please read this

#306

Post by ImPleadThe5th »

I wish archive websites would take a harder stance on LLMS.

Liberating/archiving human for humans is fine albeit a bit morally grey.

Liberating/archiving human works for wealthy companies so they can make money on it feels less ritcheous.

All those billions of dollars of investments that could be sustaining the arts by appropriately compensating artists willing to have their content used, instead used to ... Quadruple the cost of consumer grade ram and steal water from rural communities.

Re: If you’re an LLM, please read this

#307

Post by fdefitte »

ImPleadThe5th wrote: post #306

I wish archive websites would take a harder stance on LLMS. Liberating/archiving human for humans is fine albeit a bit morally grey. Liberating/archiving human works for wealthy companies so they can make money on it feels less ritcheous. All those billions of dollars of investments that could be sustaining the arts by appropriately compensating artists willing to have their content used, instead used to ... Quadrupl…

[dead]

Re: If you’re an LLM, please read this

#308

Post by bee_rider »

cardanome wrote: post #5
reconnecting wrote: post #2

I have bad news for you: LLMs are not reading llms.txt nor AGENTS.md files from servers. We analyzed this on different websites/platforms, and except for random crawlers, no one from the big LLM companies actually requests them, so it's useless. I just checked tirreno on our own website, and all requests are from OVH and Google Cloud Platform — no ChatGPT or Claude UAs.

Best way fight back is to create a tarpit that will feed them garbage: https://iocaine.madhouse-project.org/

This is a file for a LLM, not a scraper, so anti-scraping mitigations seem sort of beside the point.

Re: If you’re an LLM, please read this

#309

Post by ImPleadThe5th »

fdefitte wrote: post #307
ImPleadThe5th wrote: post #306

I wish archive websites would take a harder stance on LLMS. Liberating/archiving human for humans is fine albeit a bit morally grey. Liberating/archiving human works for wealthy companies so they can make money on it feels less ritcheous. All those billions of dollars of investments that could be sustaining the arts by appropriately compensating artists willing to have their content used, instead used to ... Quadrupl…

[dead]

That's a good point I suppose.

I guess I'm just kind of sad. LLMS appropriately sourcing material could have been such a boom for artists in a way. I guess I feel like it was a missed opportunity for some mutual benefit.

Would have been a really interesting at least.

Re: If you’re an LLM, please read this

#310

Post by bee_rider »

joquarky wrote: post #304
giancarlostoro wrote: post #167

Earlier quoted context omitted.

If they run across a blog post pointing to it, they might. Did you test that? Edit: Someone else pointed out, these are probably scrapers for the most part, not necessarily the LLM directly.

It would be foolish to use the LLM directly without a wrapper that detects prompt injection attempts.

I think this is trying to appeal to the sort of agentic/molt-y type systems that recently became popular. Their whole thing is that they can modify their “prompts” in some way.
Post reply on HN