Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

111–120 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#111

Post by zzo38computer »

okasaki wrote: post #52

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either coopera…

I also do not like Cloudflare.

If the crawlers were aware of these archive files, and would be willing to use it, then that would help, but it isn't. (It would also help to know which dynamic files are worthless for archiving and mirroring, but they will often ignore that.)

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#113

Post by p3rls »

bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

At this point I'd take a thermostat that can read when my dashboard starts getting heated (always the same culprits causing these same server spikes) and flicks attack mode on for cloudflare.... it's so ridiculous trying to run anything that's not a wordpress these days

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#114

Post by pjc50 »

renewiltord wrote: post #86
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

People are doing exactly that. And then other people who want to use the website are asking why they get blocked by false positives.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#115

Post by 1vuio0pswjnm7 »

AbortedLaunch wrote: post #102
1vuio0pswjnm7 wrote: post #96

Earlier quoted context omitted.

"It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished [sic] conventions (respecting robots.txt, using proper UA string, rate limiting, whatever)." How does "proper UA string" solve this "blowing up websites" problem The only thing that matters with respect to the "blowing up websites" problem is rate-limiting, i.e., behaviour "Shitty crawlers" are a nui…

Some of these crawlers appear to be designed to avoid rate limiting based on IP. I regularly see millions of unique ips doing strange requests, each just one or at most a few per day. When a response contains a unique redirect I often see a geographically distinct address fetching the destination.

"I regularly see millions of unique ips doing strange requests, each just one or at most a few per day."

How would UA string help

For example, a crawler making "strange" requests can send _any_ UA string, and a crawler doing "normal" requests can also send _any_ UA string.

The "doing requests" is what I refer to as "behaviour"

A website operator might think "Crawlers making strange requests send UA string X but not Y"

Let's assume the "strange" requests cause a "website load" problem^1

Then a crawler, or any www user, makes a "normal" request and sends UA string X; the operator blocks or redirects the request, unnecessarily

Then a crawler makes "strange" request and sends UA string Y; the operator allows the request and the website "blows up"

What matters for the "blowing up websites" problem^1 is behaviour, not UA string

1. The article's title calls it the "blowing up websites" problem, but the article text calls it a problem with "website load". As always the details are missing. For example, what is the "load" at issue. Is it TCP connections or HTTP requests. What number of simultaneous connections and/or requests per second is acceptable, what number is not unacceptable. Again, behaviour is the issue, not UA string

The acceptable numbers need to be published; for example, see documentation for "web APIs"

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#116

Post by Nextgrid »

integralid wrote: post #105
EgregiousCube wrote: post #42

Earlier quoted context omitted.

Under what law? It's interesting because these are sites that host content for the purpose of providing it to anonymous network users. ebay won a case against a scraper back in 2000 by claiming that the server load was harming them, but that reasoning was later overturned because it's difficult to say that server load is actual harm. ebay was in the same condition before and after a scrape. Maybe some civil lawsuit a…

On the other hand, DDoS attacks are pretty clearly on the illegal side. I wonder how this would play out in practice.

Intention plays a part. (D)DoS is intentionally done to make a website unavailable to legitimate users. Scraping may do this as a side-effect (if you are incompetent and/or use the "cloud"), but isn't the intention.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#117

Post by Nextgrid »

ben_w wrote: post #101
renewiltord wrote: post #94

Earlier quoted context omitted.

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?

But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others.

If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it.

(also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#118

Post by Nextgrid »

nromiun wrote: post #11
internet_points wrote: post #10

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

CDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well. Anubis's creator says the same thing: > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Clou…

Alternatively, go-away is also an option: https://git.gammaspectra.live/git/go-away, especially if you don't want cringy branding.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#119

Post by encrypted_bird »

timsh wrote: post #33

A bit off-topic but wtf is this preview image of a spider in the eye? It’s even worse than the clickbait title of this post. I think this should be considered bad practice.

I fully agree, and speaking as someone macroinsectophobia (fear of large or many insect (or insect-like) creatures), seeing it really makes me uncomfortable. It isn't enough to send me into panic mode or anything, but damn if it doesn't freak me out.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#120

Post by shepherdjerred »

bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

ah, you're the one who stopped me from being jerred@shepherd.com!
Post reply on HN