Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

211–220 of 279 posts

Re: We can't have nice things because of AI scrapers

#211

Post by nottorp »

Okay, it's been established that "AI" crawlers are a pest. One of the reasons being that they don't actually run any "AI", that would be too expensive.

You can't ban by user agent because that will only catch the few crawlers that are actually honest about it.

Aren't there rate limiting solutions built into at least some web servers? At least if you control your own web server, can't you do it through some reverse proxy?

Cut off IPs that make more than NN requests in a minute? Require some kind of login to allow more, if you do have endpoints that are designed to be bulk hit?

There should be ready made solutions for this still. In spite of the current answer being "lulz it's too hard, just use cloudflare".

Re: We can't have nice things because of AI scrapers

#212

Post by olyjohn »

chrismorgan wrote: post #186
trollbridge wrote: post #48

Earlier quoted context omitted.

So what you’re saying is that a CAPTCHA cannot actually T C and H A.

Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limitin…

How does a human abuse a website, and how does a CAPTCHA stop a human from abusing a website when it's designed to let a human in? If it doesn't stop humans from abusing the site, then it must stop... computers from abusing the site. And it stops computers by using the CAPTCHA to tell apart a human and a computer? Am I wrong here?

Re: We can't have nice things because of AI scrapers

#213

Post by nicbou »

akuyou wrote: post #138

AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infin…

Then they use the data to deny you traffic. AI summaries are wrecking the independent web. Losing more than half or more of your traffic was pretty common in 2025. It’s killing the economics of sharing hard-earned information.

So we are spending more resources reaching a lot less people, because a few big companies are capturing the value for their shareholders.

And that’s while they’re still haemorrhaging money! Once they fully establish their monopoly and kill the open web, the enshittification will begin.

Re: We can't have nice things because of AI scrapers

#214

Post by zimpenfish »

dpkirchner wrote: post #143
chao- wrote: post #116

Earlier quoted context omitted.

Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?

Nor Safari on iOS.

Works fine on my iOS Safari - maybe there's some extension that's tickling it just the wrong way?

Re: We can't have nice things because of AI scrapers

#215

Post by zimpenfish »

doublerabbit wrote: post #115
zimpenfish wrote: post #66

Earlier quoted context omitted.

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org

Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth. I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior. I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?

> You've wasted 500Mb of bandwidth.

Yep, it sucks, but on the positive side, I'm feeding 500Mb of garbage into them every day and that feels like enough of a small win for me.

> My monthly b/w cost is now around 20-30Gb a month given to scrapers [...] 1-2Gb a month

That definitely sucks.

> Do I go offline or let it continue?

Might be time to start blocking entire IP ranges and ASNs and see if that helps.

Re: We can't have nice things because of AI scrapers

#216

Post by levleontiev »

I am terribly sorry for self-advertising, but:

I am just now busy building a solution: self-hosted sophisticated rate-limiting.

More complex than nginx, more private than cloudfare. Please joint the waitlist if you want to morally support me ;)

https://getfairvisor.com/

Re: We can't have nice things because of AI scrapers

#217

Post by jeroenhd »

everybodyknows wrote: post #166
bodantogat wrote: post #19

I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.

> residential IP pools So, is this a new profit center for sleazeball household ISPs?

"Residential proxy" is just a word for a botnet. Apps and programs come with a trojan built in that offers your device as an exit node to "monetize" their apps.

It "residential proxy" sounds a lot better when you're talking to VC investors, though.

Re: We can't have nice things because of AI scrapers

#218

Post by jeroenhd »

accrual wrote: post #101

Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...

AI scrapers already fake user agent headers, ignore robots.txt, and go through botnets to bypass firewall rules. They're not going to put out such a signal if they can help it.

Re: We can't have nice things because of AI scrapers

#219

Post by expedition32 »

danaris wrote: post #210
chii wrote: post #203

Earlier quoted context omitted.

> refusing to respect the site owners wishes should a site owner be able to discriminate between a bot visitor and a human visitor? Most do, and hence the bots treats it as a hostile environment. Of course, bots that behave badly have created this problem themselves. That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

> That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor. Well, right; that's the problem. They take up orders of magnitude more resources. They absolutely hammer the server. They don't care if your website even survives , so long as they get every single drop of data they can for training. Source: my own personal experience with them taking down my tiny brow…

This is like email were eventually 90% of it was spam and we all got spam filters.

Re: We can't have nice things because of AI scrapers

#220

Post by soco »

fartfeatures wrote: post #137
ethin wrote: post #132

Earlier quoted context omitted.

You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.

I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.

Simpler and efficient for who? I imagine some random guy vibe coding "hi chatgpt I want to scrape this and this website", getting something running, then going to LinkedIn to brag about AI. Yes I have no hard evidence for this, but I see things on LinkedIn.
Post reply on HN