Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miasma: A tool to trap AI web scrapers in an endless poison pit

github.com

Post reply on HN
Print view View original

271–276 of 276 posts

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#271

Post by [deleted] »

Twirrim wrote: post #264
madeofpalk wrote: post #8

Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.

I have no idea if it works, but Anthropic in particular spent a lot of time crawling the tar-pit[1] I had running on my domain. They were the reason I set up the tar pit in the first place, as they were at one stage averaging 5 requests per second, for days, on a blog site that probably doesn't even have a hundred pages on it. They've retrieved millions of pages of content from my tar-pit that were texts generated vi…

[deleted]

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#272

Post by account42 »

ethmarks wrote: post #132
FeepingCreature wrote: post #119

Earlier quoted context omitted.

you're totally right about not being theft, but we have a term. you used it yourself, "distributed denial of service". that's all it is. these crawlers should be kicked off the internet for abuse. people should contact the isp of origin.

Firstly, since this argument is about semantic pedantry anyways, it's just denial-of-service, not distributed denial-of-service. AI scraper requests come from centralized servers, not a botnet. Secondly, denial-of-service implies intentionality and malice that I don't think is present from AI scrapers. They cause huge problems, but only as a negligent byproduct of other goals. I think that the tragedy of the commons…

If they came from centralized servers they would be easy to block. The whole problem is that they have a seemingly endless supply of source IPs - that' means they are "distributed" in every way that matters on the Internet even if the requests are coordinated centrally.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#274

Post by FeepingCreature »

FromTheFirstIn wrote: post #268
FeepingCreature wrote: post #244

Earlier quoted context omitted.

I agree that's bad at any rate. However, I genuinely think that reading and learning without literal reproduction is not (should not be) a violation of copyright and does not (should not) require an additional grant for content that has been made publicly available. I think that regardless of whether a company is the subject or the actor.

But you’d usually have to pay for access to this copyrighted material, whether you reproduce it or not.

Yep, and Anthropic lost that case correctly. I just don't think that "you have to buy one copy" will fix anything related to AI to the satisfaction of anybody but the law.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#275

Post by xmcp123 »

101008 wrote: post #206

Based on this comment: > I definitely get this. The thing that gives me hope is that you only need to poison a very small % of content to damage AI models pretty significantly. It helps combat the mass scraping, because a significant chunk of the data they get will be useless, and its very difficult to filter it by hand It'd be great if the code returned by this project is code that doesn't work. Imagine if all these…

I just checked some of the content from miasma, and this appears to be exactly what it does.

Everything from loops that won’t end to incorrect function calls and emoji “definitions” that are both realistic and wrong.

Very impressive project tbh.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#276

Post by pmlnr »

oasisbob wrote: post #196
spiderfarmer wrote: post #192

Earlier quoted context omitted.

I'm being hit with 300 req/s 24/7 from hundreds of thousands of unique IP's from residential proxies. I can't rate limit any further without hurting the real users.

Yeah, IP-based rate limits are nearly ineffective these days.

Nginx has per server, per path limiting options.
Post reply on HN