Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

41–50 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#41

Post by ack_complete »

delfinom wrote: post #26

I run a symbol server, as in, PDB debug symbol server. Amazon's crawler and a few others love requesting the ever loving shit out of it for no obvious reason. Especially since the files are binaries. I just set a rate-limit in cloudflare because no legitimate symbol server user will ever be excessive.

I have a simple website consisting solely of static webpages pointing to a bunch of .zip binaries. Nothing dynamic, all highly cacheable. The bots are re-downloading the binaries over and over. I can see Bingbot downloading a .zip file in the logs, and then an hour later another Bingbot instance from a different IP in the same IP range downloading the same .zip file in full. These are files that were uploaded years ago and have never retroactively changed, and don't contain crawlable contents within them (executable code).

Web crawlers have been around for years, but many of the current ones are more indiscriminate and less well behaved.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#42

Post by EgregiousCube »

xrd wrote: post #28

Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites. Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?

Under what law? It's interesting because these are sites that host content for the purpose of providing it to anonymous network users. ebay won a case against a scraper back in 2000 by claiming that the server load was harming them, but that reasoning was later overturned because it's difficult to say that server load is actual harm. ebay was in the same condition before and after a scrape.

Maybe some civil lawsuit about terms of service? You'd have to prove that the scraper agreed to the terms of service. Perhaps in the future all CAPTCHAs come with a TOS click-through agreement? Or perhaps every free site will have a login wall?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#43

Post by mrweasel »

danaris wrote: post #18
rco8786 wrote: post #15

Earlier quoted context omitted.

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

Same here. I run a very small browser game (~120 weekly users currently), and until I put its Wiki (utterly uninteresting to anyone who doesn't already play the game) behind a login-wall, the bots were causing massive amounts of spurious traffic. Due to some of the Wiki's data coming live from the game through external data feeds, the deluge of bots actually managed to crash the game several times, necessitating a re…

Wikis seems to attract AI bots like crazy, especially the bad kind that will attempt any type of cache invalidation available to them.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#45

Post by mrweasel »

miohtama wrote: post #8
pm215 wrote: post #7

Earlier quoted context omitted.

Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?

Even if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.

Many AI crawlers seems to go to great length to avoid caches, not sure why.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#46

Post by mrweasel »

hereme888 wrote: post #16

I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.

They just don't need to hammer sites into the ground to do it. This wouldn't be an issue if the AI companies where a bit more respectful of their data sources, but they are not, they don't care.

All this attempting to block AI scrapers would not be an issue if they respected rate-times, knew how to back of when a server starts responding to slowly, or caching frequently visited sites. Instead some of these companies will do everything, including using residential ISPs, to ensure that they can just piledrive the website of some poor dude that's just really into lawnmowers, or the git repo of some open source developer who just want to share their work.

Very few are actually against AI-crawlers, if they showed just the tiniest amount of respect, but they don't. I think Drew Devault said it best: "Please stop externalizing your costs directly into my face"

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#49

Post by pjc50 »

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why.

AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher electricity bills, because keeping humans warm is not as profitable as a machine which directly converts electricity into stock price rises.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#50

Post by rco8786 »

muzani wrote: post #21
rco8786 wrote: post #14

OpenAI straight up DoSed a site I manage for my in-laws a few months ago.

What is it about? I'm curious what kinds of things people ask that floods sites.

The site is about a particular type of pipeline cleaning (think water/oil pipelines). I am certain that nobody was asking about this particular site or even the industry its in 15,000 times a minute 24 hours a day.

It's much more likely that their crawler is just garbage and got stuck into some kind of loop requesting my domain.

Post reply on HN