Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miasma: A tool to trap AI web scrapers in an endless poison pit

github.com

Post reply on HN
Print view View original

51–60 of 276 posts

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#51

Post by otherme123 »

raincole wrote: post #39
madeofpalk wrote: post #8

Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.

It might work against people just use their Mini Mac with OpenClaw to summarize news every morning, but it certainly won't work against Google. More centralized web ftw.

In my experience, Google (among others) plays nice. Just put "disallow: *" in your robots.txt, and they won't bother you again.

My current problem is OpenAI, that scans massively ignoring every limit, 426, 444 and whatever you throw at them, and botnets from East Asia, using one IP per scrap, but thousands of IPs.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#52

Post by DaiPlusPlus »

arbol wrote: post #45
snehesht wrote: post #12

Why not simply blacklist or rate limit those bot IP’s ?

The AI companies are using virtually unlimited "clean" residential IPs so this is not a valid strategy.

How? They run their scraping and training infrastructure - and models themselves - from within those “AI datacenters”[1] we hear about in the news - and not proxying through end-users’ own pipes.

[1]: in quotes, because I dislike the term, because it’s immaterial whether or not an ugly block of concrete out in the sticks is housing LLM hardware - or good ol’ fashioned colo racks.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#53

Post by volemo »

falcor84 wrote: post #40
spiderfarmer wrote: post #32

Earlier quoted context omitted.

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

That really depends, but the quick answer is that according to our human social contract, we'd just ask "how many can I take?". Until now, the only real tool to limit scrapers has been throttling, but I don't see any reason for there not to be a similar conversational social contract between machines.

Isn’t robots.txt such a “social contract between machines”? But AI scrapers couldn’t care less.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#55

Post by hexage1814 »

raincole wrote: post #39
madeofpalk wrote: post #8

Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.

It might work against people just use their Mini Mac with OpenClaw to summarize news every morning, but it certainly won't work against Google. More centralized web ftw.

It also probably won't work if the person actually wants your content and is checking if the thing they scraped actually makes sense or it just noise. Like, none of these are new things. Site owners send junk/fake data to webscrapers since web scraping was invented.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#58

Post by pixl97 »

snehesht wrote: post #38
aduwah wrote: post #21

Earlier quoted context omitted.

There are way too many to do that

True, most of the blacklists systems today aren’t realtime like Amazon WAF or Cloudflare. We need a Crawler blacklist that can in realtime stream list deltas to centralized list and local dbs can pull changes. Verified domains can push suspected bot ips, where this engine would run heuristics to see if there is a patters across data sources and issue a temporary block with exponential TTL. There are many problems to…

In the real-time spam market the lists worked well with honest groups for a bit, but started falling apart when once good lists get taken over by actors that realize they can use their position to make more money. It's a really difficult trap to avoid.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#59

Post by throwaway613746 »

drfloyd51 wrote: post #44
spiderfarmer wrote: post #32

Earlier quoted context omitted.

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

Odd thing about cookies… they disappear after one serving. Websites are an endless stream of cookies. The analogy doesn’t hold.

[dead]

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#60

Post by ghywertelling »

drfloyd51 wrote: post #44
spiderfarmer wrote: post #32

Earlier quoted context omitted.

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

Odd thing about cookies… they disappear after one serving. Websites are an endless stream of cookies. The analogy doesn’t hold.

If copying content from harddrive to another is theft, then so is DNA copying itself.

Everything is a Remix culture. We should promote remix culture rather than hamper it.

Everything is a Remix (Original Series) https://youtu.be/nJPERZDfyWc

Post reply on HN