AI crawlers need to be more respectful • Hacker News Board

Re: AI crawlers need to be more respectful

#81

Post by Joel_Mckay » Thu, Jul 25, 2024, 9:43 PM UTC

reaperman wrote: post #44

Joel_Mckay wrote: post #28

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of…

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.

The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know...

And Reddit has definitely become more proactive about scrapers. ;-)

Re: AI crawlers need to be more respectful

#82

Post by bo1024 » Thu, Jul 25, 2024, 10:05 PM UTC

int3 wrote: post #20

Shouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place

They say this:

> We have IP-based rate limiting in place for many of our endpoints, however these crawlers are coming from a large number of IP addresses, so our rate limiting is not effective.

Do you have something else in mind? Just shut down the whole site after a certain limit?

Re: AI crawlers need to be more respectful

#83

Post by Joel_Mckay » Thu, Jul 25, 2024, 10:05 PM UTC

Atotalnoob wrote: post #46

Joel_Mckay wrote: post #28

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of…

AI/ML has created a race to suck all the data on the internet regardless of copyright or status and use it. OpenAI introduced gptbot in August 2023… they already took everything

Unlikely, site-generator hosts are still happily providing a limitless supply of remixed well-structured nonsense, random images with noise, and valid links to popular sites.

In this case, they showed up to the data buffet long after it went rotten due to SEO.

Have a nice day =3

Re: AI crawlers need to be more respectful

#84

Post by bastawhiz » Thu, Jul 25, 2024, 10:19 PM UTC

Saris wrote: post #77

gnfargbl wrote: post #69

Earlier quoted context omitted.

10TB/day is, roughly, a single saturated 1Gbit link. In technical bandwidth terms, that is the square root of fuck all . The crazy thing here is that the target site is content to pay ~$700 for the volume of traffic that you can move through a single teeny-tiny included-at-no-extra-charge cat5 Ethernet link in one single day. And apparently, they're going to continue doing so.

Yeah that's an insane price for bandwidth, they need to move providers ASAP if that's the kind of fees they get.

Hosting documentation shouldn't need that much bandwidth. It's text and zip files full of text. Without bots, that's a very small cost even if the bytes are relatively costly.

Re: AI crawlers need to be more respectful

#85

Post by bastawhiz » Thu, Jul 25, 2024, 10:22 PM UTC

mschuster91 wrote: post #19

simonw wrote: post #10

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.

The more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day. If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.

I've been running guthib.mattbasta.workers.dev for years, and in the past few months it's hit the free limit for requests every day. Some AI company is filling their corpus with exactly that: infinite garbage.

Re: AI crawlers need to be more respectful

#86

Post by croemer » Thu, Jul 25, 2024, 10:26 PM UTC

Joel_Mckay wrote: post #76

immibis wrote: post #55

Earlier quoted context omitted.

It's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when tha…

Really depends on the use-case, but yeah the response happens in a proportional manner. We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs. And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along wit…

Why are you ending all your messages with =3 ?

Re: AI crawlers need to be more respectful

#87

Post by croemer » Thu, Jul 25, 2024, 10:29 PM UTC

Joel_Mckay wrote: post #74

pests wrote: post #45

Earlier quoted context omitted.

6 hours a minute? Are you joking? Click around a few times on any of your sites and looks like I'll be banned? Multi tasking? Opening multiple interesting links? Like what.

Generally for sites: 1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.) 2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading) 3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban 4. if you trip a spider trap, than you get a 5 d…

Who is "we"?

Re: AI crawlers need to be more respectful

#88

Post by Joel_Mckay » Thu, Jul 25, 2024, 10:37 PM UTC

croemer wrote: post #86

Joel_Mckay wrote: post #76

Earlier quoted context omitted.

Really depends on the use-case, but yeah the response happens in a proportional manner. We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs. And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along wit…

Why are you ending all your messages with =3 ?

https://www.jpl.nasa.gov/images/pia22092-arp-142-the-penguin...

Don't worry about it friend =3

Re: AI crawlers need to be more respectful

#89

Post by Joel_Mckay » Thu, Jul 25, 2024, 10:38 PM UTC

croemer wrote: post #87

Joel_Mckay wrote: post #74

Earlier quoted context omitted.

Generally for sites: 1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.) 2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading) 3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban 4. if you trip a spider trap, than you get a 5 d…

Who is "we"?

Plural of the deployment team admins.

Don't worry about it friend =3

Re: AI crawlers need to be more respectful

#90

Post by echoangle » Thu, Jul 25, 2024, 10:39 PM UTC

OtherShrezzing wrote: post #41

stusmall wrote: post #34

Earlier quoted context omitted.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

Naming names isn't really required. The hosts have a $5,000 bandwidth fee, but so do the consumers. There's maybe 10 companies with the financial & compute resources to let a $5,000-per-month-per-website bug run rampant before taking the harvesting service offline. Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites…

You know you can hit the data rate they were complaining about by using a residential fiber connection, right? 10 TB per day is about 1 Gigabit continuous if I’m not mistaken. There are probably millions of people that could to this if they wanted to.