Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

101–110 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#101

Post by ben_w »

renewiltord wrote: post #94
PaulDavisThe1st wrote: post #91

Earlier quoted context omitted.

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ...…

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#102

Post by AbortedLaunch »

1vuio0pswjnm7 wrote: post #96
rnhmjoj wrote: post #54

Earlier quoted context omitted.

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

"It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished [sic] conventions (respecting robots.txt, using proper UA string, rate limiting, whatever)." How does "proper UA string" solve this "blowing up websites" problem The only thing that matters with respect to the "blowing up websites" problem is rate-limiting, i.e., behaviour "Shitty crawlers" are a nui…

Some of these crawlers appear to be designed to avoid rate limiting based on IP. I regularly see millions of unique ips doing strange requests, each just one or at most a few per day. When a response contains a unique redirect I often see a geographically distinct address fetching the destination.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#103

Post by IT4MD »

pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

>AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet

10/10. No notes.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#104

Post by superkuh »

autoexec wrote: post #82
superkuh wrote: post #67

Earlier quoted context omitted.

This isn't AI damaging anything. This is corporations damaging things. Same as it ever was. No need for scifi non-human persons when legal corporate persons exist. They latch on to whatever big new thing in tech that people don't understand which comes along and brand themselves with it and cause damage trying to make money; even if they mostly fail at it. And for most actual humans they only ever see or interact wit…

> This isn't AI damaging anything. This is corporations damaging things. This is corporations damaging things because of AI. Corporations will damage things for other reasons too but the only reason they are breaking the internet in this way, at this time, is because of AI. I think the "AI doesn't kill websites, corporations kill websites" argument is as flawed as the "Guns don't kill people, people kill people" argu…

Correct. It's a good, legitimate argument in both contexts. I use both local AI and local firearms as a human person and I am not doing, and have not done, damage to anyone. The tools aren't the problem.

The problem in this case is the near complete protection from legal liability that corporate structures give to the people behaving badly. Like how Coca Cola can get away with killing people (https://prospect.org/features/coca-cola-killings/) but a person can't, if you want to keep the firearms analogy going. But it's a bad analogy because the firearms as tool actually at least are involved in the bad (and good) actions. AI itself isn't even involved in the HTTP requests and probably isn't even running on the same premises.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#105

Post by integralid »

EgregiousCube wrote: post #42
xrd wrote: post #28

Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites. Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?

Under what law? It's interesting because these are sites that host content for the purpose of providing it to anonymous network users. ebay won a case against a scraper back in 2000 by claiming that the server load was harming them, but that reasoning was later overturned because it's difficult to say that server load is actual harm. ebay was in the same condition before and after a scrape. Maybe some civil lawsuit a…

On the other hand, DDoS attacks are pretty clearly on the illegal side. I wonder how this would play out in practice.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#106

Post by bakugo »

GuB-42 wrote: post #79
bakugo wrote: post #24

Earlier quoted context omitted.

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barel…

Surely there are solutions more subtle than blocking 80% of the world population...

I should've made it clear that it's not a block rule, just a challenge rule. Those people can still access the website, they just have to go through the "checking your browser" page that you're probably familiar with.

As I said, you can just enable that for everyone and be done with it, but with a custom rule, you can avoid showing it to people that are unlikely to be bots.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#107

Post by majkinetor »

spinningslate wrote: post #80
majkinetor wrote: post #71

Earlier quoted context omitted.

Obaying robots.txt can not be enforced. Even if one country makes laws about it, another one will have 0 fucks to give.

It was never intended to be "enforced": > The standard, developed in 1994, relies on voluntary compliance [0] It was conceived in a world with an expectation of collectively respectful behaviour: specifically that search crawlers could swamp "average Joe's" site but shouldn't . We're in a different world now but companies still have a choice. Some do still respect it... and then there's Meta, OpenAI and such. Communi…

Thanks for the info. However people seem to think that robots.txt will protect them while it was created for another world as you nicelly stated. I guess Nepenthes like tools will be more common in the future, now that tragedy of commons entered digital domain.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#108

Post by greatgib »

That's the moment that you remember that years ago in self hosted you could have sustained millions request per second on a single low end server for quite nothing.

But now you are "on the cloud", with lambdas because "who cares" and hiring a proper part-time sysadmin is too complicated and so now you are pounded with crazy costs for moderate loads...

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#109

Post by tadfisher »

PaulDavisThe1st wrote: post #88
kjkjadksj wrote: post #84

Earlier quoted context omitted.

People have been complaining about these crawlers for years well before AI

The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.

Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free.

[0]: https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disas...

[1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#110

Post by PaulDavisThe1st »

tadfisher wrote: post #109
PaulDavisThe1st wrote: post #88

Earlier quoted context omitted.

The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.

Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free. [0]: https://dr…

> Tragedy of the commons.

No, because there is no such thing, at least not as understood by Garrett Hardin, who put forward the phrase.

Commons fail when selfish, greedy people subvert or destroy the governance structures that help control them. If those governance structures exist (and they do for all historical commons) and continue to exist, the commons suffers no tragedy.

This recent slide deck talks about Ostrom's ideas on this, which even Hardin eventually conceded were correct, and that his diagnosis of a "tragedy of the commons" does not actually describe the historical processes by which commons are abused.

https://dougwebb.site/slides/commons

That said ... arguably there is a problem here with a "commons" that does in fact lack any real governance structure.

Post reply on HN