Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

131–140 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#131

Post by defrost »

BrenBarn wrote: post #130

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publ…

> But How?

Typically by hiring them .. although their provenance might be sketchy..

eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man...

^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar device mapping project).

More legally there are many "free software" deals that offer services for people via installed software that comes with a side order of background web crawling in the fine print of the WALL-O'-TEXT Terms Of Uses agreement.

Enterprising middle people gather up bots and offer them for hire to web crawlers, large scale companies will farm their own bots via their existing user base.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#132

Post by AbortedLaunch »

1vuio0pswjnm7 wrote: post #115
AbortedLaunch wrote: post #102

Earlier quoted context omitted.

Some of these crawlers appear to be designed to avoid rate limiting based on IP. I regularly see millions of unique ips doing strange requests, each just one or at most a few per day. When a response contains a unique redirect I often see a geographically distinct address fetching the destination.

"I regularly see millions of unique ips doing strange requests, each just one or at most a few per day." How would UA string help For example, a crawler making "strange" requests can send _any_ UA string, and a crawler doing "normal" requests can also send _any_ UA string. The "doing requests" is what I refer to as "behaviour" A website operator might think "Crawlers making strange requests send UA string X but not Y…

I do not make any point on UA-strings, just on the difficulty of rate limiting.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#133

Post by latexr »

Nextgrid wrote: post #117
ben_w wrote: post #101

Earlier quoted context omitted.

Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?

But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of t…

> But what's the difference between one user making 900k hits and 900k different users making one hit?

What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people.

Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot.

> makes me suspect there's substantial incompetence in the solutions deployed to serve them

So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#134

Post by renewiltord »

msgodel wrote: post #129
renewiltord wrote: post #94

Earlier quoted context omitted.

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages. I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to s…

Seems fine to me. Same as ads. If you don’t want to send content with ads which I will render without ads don’t send. That ended some businesses and made others paywall.

Such is life.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#135

Post by BrenBarn »

defrost wrote: post #131
BrenBarn wrote: post #130

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publ…

> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar…

Okay, but are OpenAI and Meta straight up buying botnets on the black market?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#136

Post by defrost »

BrenBarn wrote: post #135
defrost wrote: post #131

Earlier quoted context omitted.

> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar…

Okay, but are OpenAI and Meta straight up buying botnets on the black market?

Unlikely.

There are many ways, at their scale they (Meta at least) probably have edge servers in ISP's across the planet and can easily mix their crawlers with residential IP addresses rotationally assigned by domestic ISP's they co-mingle with.

Or some other way, legal but somewhat obfuscated.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#137

Post by subscribed »

bwb wrote: post #85
weaksauce wrote: post #64

Earlier quoted context omitted.

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

Got a good link to something on github that does this? I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.

I did it manually and got fail2ban to read the access log anyway.

The it's the permanent iptables rule, but could be CF API call as well.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#138

Post by sznio »

rnhmjoj wrote: post #54
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

I strongly believe that AI companies are running a DDOS attack on the open web. Making websites go down aligns with their intetests: it removes training data that competitors could use, and it removes sources for humans to browse, making us even more reliant on chatbots to find anything.

If it was crap coding, then the bots wouldn't have so many mechanisms to circumvent blocks. Once you block the OpenAI IP ranges, they start using residential proxies. Once you block their UA strings, they start impersonating other crawlers or browsers.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#139

Post by bwb »

shepherdjerred wrote: post #120
bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

ah, you're the one who stopped me from being jerred@shepherd.com!

hah eh?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#140

Post by diggan »

BrenBarn wrote: post #135
defrost wrote: post #131

Earlier quoted context omitted.

> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar…

Okay, but are OpenAI and Meta straight up buying botnets on the black market?

OpenAI i'm not so sure about, but since Meta already got caught downloading copyrighted material to train LLMs, I think it isn't far fetched for them to also use borderline illegal methods for acquiring IPs to use.
Post reply on HN