AI crawlers need to be more respectful • Hacker News Board

Re: AI crawlers need to be more respectful

#41

Post by OtherShrezzing » Thu, Jul 25, 2024, 8:42 PM UTC

stusmall wrote: post #34

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

Naming names isn't really required. The hosts have a $5,000 bandwidth fee, but so do the consumers. There's maybe 10 companies with the financial & compute resources to let a $5,000-per-month-per-website bug run rampant before taking the harvesting service offline.

Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites with that crawler.

Re: AI crawlers need to be more respectful

#42

Post by energy123 » Thu, Jul 25, 2024, 8:44 PM UTC

OtherShrezzing wrote: post #41

stusmall wrote: post #34

Earlier quoted context omitted.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

Naming names isn't really required. The hosts have a $5,000 bandwidth fee, but so do the consumers. There's maybe 10 companies with the financial & compute resources to let a $5,000-per-month-per-website bug run rampant before taking the harvesting service offline. Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites…

I thought data egress is much more expensive than downloading it?

Re: AI crawlers need to be more respectful

#43

Post by Joel_Mckay » Thu, Jul 25, 2024, 8:45 PM UTC

stusmall wrote: post #34

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

It is 100% likely a cloud provider IP range.

They are a persistent source of spam email servers, scrapers, and bot probes.

The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists.

It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, Google, and Baidu.

Have a wonderful day, =3

Re: AI crawlers need to be more respectful

#44

Post by reaperman » Thu, Jul 25, 2024, 8:45 PM UTC

Joel_Mckay wrote: post #28

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of…

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”.

Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.

Re: AI crawlers need to be more respectful

#45

Post by pests » Thu, Jul 25, 2024, 8:47 PM UTC

Joel_Mckay wrote: post #36

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3

6 hours a minute? Are you joking?

Click around a few times on any of your sites and looks like I'll be banned?

Multi tasking? Opening multiple interesting links?

Like what.

Re: AI crawlers need to be more respectful

#46

Post by Atotalnoob » Thu, Jul 25, 2024, 8:47 PM UTC

Joel_Mckay wrote: post #28

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of…

AI/ML has created a race to suck all the data on the internet regardless of copyright or status and use it.

OpenAI introduced gptbot in August 2023… they already took everything

Re: AI crawlers need to be more respectful

#47

Post by Faaak » Thu, Jul 25, 2024, 8:48 PM UTC

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

5000$ for 73TiB seems excessive tough? Some EU cloud providers would price it 10x cheaper

Re: AI crawlers need to be more respectful

#48

Post by immibis » Thu, Jul 25, 2024, 8:49 PM UTC

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

That's also $73 worth of bandwidth on another server host. Please stop using extremely expensive hosts and then blaming other people for the consequences of your decision.

Re: AI crawlers need to be more respectful

#49

Post by immibis » Thu, Jul 25, 2024, 8:49 PM UTC

Faaak wrote: post #47

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

5000$ for 73TiB seems excessive tough? Some EU cloud providers would price it 10x cheaper

Hetzner is $1 per TB and that's only if they decide your overall usage is excessive and needs to be billed for.

Re: AI crawlers need to be more respectful

#50

Post by mikae1 » Thu, Jul 25, 2024, 8:49 PM UTC

HellPot: https://github.com/yunginnanet/HellPot

> Clients (hopefully bots) that disregard robots.txt and connect to your instance of HellPot will suffer eternal consequences. HellPot will send an infinite stream of data that is just close enough to being a real website that they might just stick around until their soul is ripped apart and they cease to exist.