Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

81–90 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#81

Post by davidfischer »

internet_points wrote: post #10

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

My employer, Read the Docs, has a blog on the subject (https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill.

We've done a few things since then:

- We already had very generous rate limiting rules by IP (~4 hits/second sustained) but some of the crawlers used thousands of IPs. Cloudflare has a list that they update of AI crawler bots (https://developers.cloudflare.com/bots/additional-configurat...). We're using this list to block these bots and any new bots that get added to the list.

- We have more aggressive rate limiting rules by ASN on common hosting providers (eg. AWS, GCP, Azure) which also hits a lot of these bots.

- We are considering using the AI crawler list to rate limit by user agent in addition to rate limiting by IP. This will allow well behaved AI crawlers while blocking the badly behaved ones. We aren't against the crawlers generally.

- We now have alert rules that alert us when we get a certain amount of traffic (~50k uncached reqs/min sustained). This is basically always some new bot cranked to the max and usually an AI crawler. We get this ~monthly or so and we just ban them.

Auto-scaling made our infra good enough where we don't even notice big traffic spikes. However, the downside of that is that the AI crawlers were hammering us without causing anything noticeable. Being smart with rate limiting helps a lot.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#82

Post by autoexec »

superkuh wrote: post #67
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

This isn't AI damaging anything. This is corporations damaging things. Same as it ever was. No need for scifi non-human persons when legal corporate persons exist. They latch on to whatever big new thing in tech that people don't understand which comes along and brand themselves with it and cause damage trying to make money; even if they mostly fail at it. And for most actual humans they only ever see or interact wit…

> This isn't AI damaging anything. This is corporations damaging things.

This is corporations damaging things because of AI. Corporations will damage things for other reasons too but the only reason they are breaking the internet in this way, at this time, is because of AI.

I think the "AI doesn't kill websites, corporations kill websites" argument is as flawed as the "Guns don't kill people, people kill people" argument.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#83

Post by sumtechguy »

GuB-42 wrote: post #79
bakugo wrote: post #24

Earlier quoted context omitted.

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barel…

Surely there are solutions more subtle than blocking 80% of the world population...

is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'.

It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#84

Post by kjkjadksj »

mostlysimilar wrote: post #63
rnhmjoj wrote: post #54

Earlier quoted context omitted.

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

But it didn't, and it's happening now, because of AI.

People have been complaining about these crawlers for years well before AI

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#85

Post by bwb »

weaksauce wrote: post #64
bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

Got a good link to something on github that does this?

I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#86

Post by renewiltord »

pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way.

Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#87

Post by discardedrefuse »

lionkor wrote: post #75
mediumsmart wrote: post #36

Earlier quoted context omitted.

so the fancy AI agent will have to get really fancy and mimic human traffic and all is good until the server heats up from all those separate human trafficionados - then what?

Nobody is forcing anyone to share their knowledge. What then? Dead internet.

Absolutely yes. I guarantee you these megacorps are betting on a future where the open internet has been completely obliterated. And the only way to participate online is thru their portal; where everything you do feeds back into their AI. Because that is the only way to acquire fresh food for their beast.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#88

Post by PaulDavisThe1st »

kjkjadksj wrote: post #84
mostlysimilar wrote: post #63

Earlier quoted context omitted.

But it didn't, and it's happening now, because of AI.

People have been complaining about these crawlers for years well before AI

The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#89

Post by GuB-42 »

sumtechguy wrote: post #83
GuB-42 wrote: post #79

Earlier quoted context omitted.

Surely there are solutions more subtle than blocking 80% of the world population...

is there an http code for 'hey I gave you this already 10 times. This is a you problem not a me problem I refuse to give you another copy'. It also sounds like there is an opportunity to sell scraped data to these companies. Instead of 10 crawlers we get one crawler and they just resell/give it away. More honey pots doesnt really fix the root cause (which is greed).

> is there an http code for 'hey I gave you this already 10 times.

429 Too Many Requests

> This is a you problem not a me problem

That's the "4" in "429"

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#90

Post by nucleardog »

ajsnigrutin wrote: post #57
pjc50 wrote: post #51

Earlier quoted context omitted.

How would you suggest that such users prove they're not a crawler?

Why would they have to? What's wrong with crawlers? That's how google finds you, and people find you on google. Just put some sensible request limits per hour per ip, and be done.

> Just put some sensible request limits per hour per ip, and be done.

I have no personal experience, but probably worth reading like... any of the comments where people are complaining about these crawlers.

Claims are that they're: ignoring robots.txt; sending fake User-Agent headers; they're crawling from multiple IPs; when blocked they will use residential proxies.

People who have deployed Anubis to try and address this include: Linux Kernel Mailing List, FreeBSD, Arch Linux, NixOS, Proxmox, Gnome, Wine, FFMPEG, FreeDesktop, Gitea, Marginalia, FreeCAD, ReactOS, Duke University, The United Nations (UNESCO)...

I'm relatively certain if this were as simple as "just set a sensible rate limit and the crawlers will stop DDOS'ing your site" one person at one of these organizations would have figured that out by now. I don't think they're all doing it because they really love anime catgirls.

Post reply on HN