We can't have nice things because of AI scrapers

Post by tdeck » Wed, Jan 14, 2026, 3:15 PM UTC

adrianwaj wrote: post #172

Look no further than x402 micropayments as both the solution and opportunity here. And then a way to return a portion to humans. These AI companies are loaded too (maybe not the long-tail as yet) and the crypto ecosystem is mature. Come one, come all. Make money. Need a Wordpress plugin to start the ball rolling and provide ping endpoints for the AI companies to leach from. They can pay to get those pings too. Give t…

Post by mattdanger » Wed, Jan 14, 2026, 3:59 PM UTC

devhouse wrote: post #102

random idea, instead of blocking scrapers, maybe detect them (via user-agent, request patterns, ignoring robots.txt) and serve them garbage data wrapped in dad jokes. if (isSuspiciousScraper(req)) { return res.json({ data: getDadJoke(), artist: "Rick Astley", // always album: "Never Gonna Give You Up" }); }

Post by expedition32 » Wed, Jan 14, 2026, 4:09 PM UTC

chii wrote: post #203

toofy wrote: post #150

Earlier quoted context omitted.

> The bots have a different model -- they assume that the website is adversarially "hiding" its content. this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem. a bot should not consider that “adversarial”

> refusing to respect the site owners wishes should a site owner be able to discriminate between a bot visitor and a human visitor? Most do, and hence the bots treats it as a hostile environment. Of course, bots that behave badly have created this problem themselves. That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

Post by zzzeek » Wed, Jan 14, 2026, 5:22 PM UTC

zimpenfish wrote: post #66

zzzeek wrote: post #47

Earlier quoted context omitted.

Citation needed

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org

Post by lxgr » Wed, Jan 14, 2026, 5:30 PM UTC

jijijijij wrote: post #226

akuyou wrote: post #138

AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infin…

My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are ze…