Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers need to be more respectful

about.readthedocs.com

Post reply on HN
Print view View original

51–60 of 128 posts

Re: AI crawlers need to be more respectful

#51

Post by everforward »

I'm curious if there's a point where a crawler is misconfigured so badly that it becomes a violation of the CFAA by nature of recklessness.

They say a single crawler downloaded 73TB of zipped HTML files in a month. That averages out to ~29 MB/s of traffic, every second, for an entire month.

Averaging 30 megabytes a second of traffic for a month is crossing into reckless territory. I don't think any sane engineer would call that normal or healthy for scraping a site like ReadTheDocs; Twitter/Facebook/LinkedIn/etc, sure, but not ReadTheDocs.

To me, this crosses into "recklessly negligent" territory, and I think should come with government fines for the company that did it. Scraping is totally fine to me, but it needs to be done either a) at a pace that will not impact the provider (read: slowly), or b) with some kind of prior agreement that the provider is accepting responsibility to provide enough capacity.

While I agree that putting content out into the public means it can be scraped, I don't think that necessarily implies scrapers can do their thing at whatever rate they want. As a provider, there's very little difference to me between getting DDoSed and getting scraped to death; both ruin the experience for users.

Re: AI crawlers need to be more respectful

#52

Post by xcv123 »

Joel_Mckay wrote: post #36
simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3

> 73TB is a fair amount to have on a cloud

From the article:

"This was a bug in their crawler that was causing it to download the same files over and over again."

Re: AI crawlers need to be more respectful

#53

Post by morkalork »

PaulHoule wrote: post #27
corbet wrote: post #21

Earlier quoted context omitted.

I think there are a thousand wannabe companies all trying to suck up as much data as they can; not a sustainable situation in any way.

There was a paper about webcrawlers circa 2000 that pointed out that the vast majority of academics who ran webcrawlers never published a paper based on their work.

Sounds like all the kids who want to make a video-game and start with building a game engine.

Re: AI crawlers need to be more respectful

#55

Post by immibis »

Joel_Mckay wrote: post #43
stusmall wrote: post #34

Earlier quoted context omitted.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

It is 100% likely a cloud provider IP range. They are a persistent source of spam email servers, scrapers, and bot probes. The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists. It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, G…

It's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when that behavior occurs, and quickly uninstated when it does not.

Re: AI crawlers need to be more respectful

#56

Post by levkk »

mikae1 wrote: post #50

HellPot: https://github.com/yunginnanet/HellPot > Clients (hopefully bots) that disregard robots.txt and connect to your instance of HellPot will suffer eternal consequences. HellPot will send an infinite stream of data that is just close enough to being a real website that they might just stick around until their soul is ripped apart and they cease to exist.

This doesn't solve their bandwidth costs which is their real problem with these bots.

Re: AI crawlers need to be more respectful

#57

Post by Joel_Mckay »

DamonHD wrote: post #38
Joel_Mckay wrote: post #36

Earlier quoted context omitted.

Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3

When the bots come out of cloud services then the IPs are all over the place: it's much harder to do right these days.

Rate limiting firewalls and spider traps also work well...

There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules.

One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

Re: AI crawlers need to be more respectful

#58

Post by immibis »

DamonHD wrote: post #35

Not just AI: here is my current side-quest: https://www.earth.org.uk/RSS-efficiency.html Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.

Can you insert a podcast item that says "you are spamming our server - please stop"?

Re: AI crawlers need to be more respectful

#59

Post by immibis »

jsheard wrote: post #31
int3 wrote: post #20

Shouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place

The big three cloud providers (AWS/GCP/Azure) have collectively decided that you don't want to set a spending limit actually, so they simply don't let you.

The big three cloud providers are the most expensive by a factor of 10-100x, and shouldn't be used under any circumstances unless you really, really need specific features from them.
Post reply on HN