Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Miasma: A tool to trap AI web scrapers in an endless poison pit

github.com

Post reply on HN
Print view View original

31–40 of 276 posts

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#31

Post by snehesht »

xprnio wrote: post #27
snehesht wrote: post #12

Why not simply blacklist or rate limit those bot IP’s ?

If you have real traffic and bot traffic, you still need to identify which is which. On top of that, bots very likely don’t reuse the same IPs over and over again. I assume if we knew all the IPs used only by bots ahead of time, then yeah it would be simple to blacklist them. But although it’s simple in theory, the practice of identifying what to blacklist in the first place is the part that isn’t as simple

You wouldn’t permanently block them, it’s more like a rolling window.

You can use security challenges as a mechanism to identify false positives.

Sure bots can get tons of proxies for cheap, doesn’t mean you can’t block them similar to how SSH Honeypots or Spamhaus SBL work albeit temporarily.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#32

Post by spiderfarmer »

tasuki wrote: post #28

> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#33

Post by spiderfarmer »

madeofpalk wrote: post #8

Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.

There are hundreds of bots using residential proxies. That is not free. Make them pay.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#34

Post by GaggiX »

spiderfarmer wrote: post #32
tasuki wrote: post #28

> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

I will copy the supermarket and paste it somewhere else.

I'm also going to download a car.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#35

Post by aldousd666 »

This is ultimately just going to give them training material for how to avoid this crap. They'll have to up their game to get good code. The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. The bottom has always been threatening to fall out of the ads paid for eyeballs, And nobody could anticipate the trigger for the downfall. Looks like we found it.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#37

Post by aldousd666 »

aldousd666 wrote: post #35

This is ultimately just going to give them training material for how to avoid this crap. They'll have to up their game to get good code. The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. The bottom has always been threatening to fall out of the ads paid for eyeballs,…

To be clear, I mean AI is going to be the downfall of ad supported content. But let's face it. We have link farms and spam factories as a result of the ad supported content market. I think this is going to eventually do justice for users because it puts a premium on content quality that someone will want to pay a direct licensing fee to scrape for your AI bots as opposed to tricking somebody into clicking on a link and looking at an impression for something they won't buy.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#38

Post by snehesht »

aduwah wrote: post #21
snehesht wrote: post #12

Why not simply blacklist or rate limit those bot IP’s ?

There are way too many to do that

True, most of the blacklists systems today aren’t realtime like Amazon WAF or Cloudflare.

We need a Crawler blacklist that can in realtime stream list deltas to centralized list and local dbs can pull changes.

Verified domains can push suspected bot ips, where this engine would run heuristics to see if there is a patters across data sources and issue a temporary block with exponential TTL.

There are many problems to solve here, but as any OSS it will evolve over time if there is enough interest in it.

Costs of running this system will be huge though and corp sponsors may not work but individual sponsors may be incentivized as it’s helps them reduce bandwidth, compute costs related to bot traffic.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#39

Post by raincole »

madeofpalk wrote: post #8

Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.

It might work against people just use their Mini Mac with OpenClaw to summarize news every morning, but it certainly won't work against Google.

More centralized web ftw.

Re: Miasma: A tool to trap AI web scrapers in an endless poison pit

#40

Post by falcor84 »

spiderfarmer wrote: post #32
tasuki wrote: post #28

> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!

If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

That really depends, but the quick answer is that according to our human social contract, we'd just ask "how many can I take?". Until now, the only real tool to limit scrapers has been throttling, but I don't see any reason for there not to be a similar conversational social contract between machines.
Post reply on HN