Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

31–40 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#31

Post by ndriscoll »

miohtama wrote: post #8
pm215 wrote: post #7

Earlier quoted context omitted.

Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?

Even if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.

Could be serving no-cache headers? Seems like yet another problem stemming from every website being designed as if it were some dynamic application when nearly all of them are static documents. nginx doing 39k req/min to cacheable pages on an n100 is what you might call "98% idle", not "unsustainable load on web servers".

The data transfer, on the other hand, could be substantial and costly. Is it known whether these crawlers do respect caching at all? Provide If-Modified-Since/If-None-Match or anything like that?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#32

Post by [deleted] »

Leynos wrote: post #27
rco8786 wrote: post #15

Earlier quoted context omitted.

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?

[deleted]

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#34

Post by lionkor »

hereme888 wrote: post #16

I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.

The second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#36

Post by mediumsmart »

lionkor wrote: post #34
hereme888 wrote: post #16

I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.

The second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.

so the fancy AI agent will have to get really fancy and mimic human traffic and all is good until the server heats up from all those separate human trafficionados - then what?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#37

Post by hereme888 »

lionkor wrote: post #34
hereme888 wrote: post #16

I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.

The second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.

I've never ran any public-facing servers, so maybe I'm missing the experience of your frustration. But mine, as a "consumer" is wanting clean answers, like what you'd expect when asking your own employee for information.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#38

Post by jasoncartwright »

I recently, for pretty much the first time ever in 30 years of running websites, had to blanket ban crawlers. I now whitelist a few, but the rest (and all other non-UK visitors) have to pass a Cloudflare challenge [1].

AI crawlers were downloading whole pages and executing all the javascript tens of millions of times a day - hurting performance, filling logs, skewing analytics and costing too much money in Google Maps loads.

Really disappointing.

[1] https://developers.cloudflare.com/cloudflare-challenges/

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#39

Post by outside1234 »

xrd wrote: post #28

Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites. Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?

Yes. There are one set of rules for us and another set of rules for anything with more than a billion dollars.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#40

Post by ajsnigrutin »

hombre_fatal wrote: post #25
internet_points wrote: post #10

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

CloudFlare's Super Bot Fight Mode completely killed the surge in bot traffic for my large forum.

And added captchas to every user with an adblock or sensible privacy settings.
Post reply on HN