Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

121–130 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#121

Post by dwd »

rco8786 wrote: post #50
muzani wrote: post #21

Earlier quoted context omitted.

What is it about? I'm curious what kinds of things people ask that floods sites.

The site is about a particular type of pipeline cleaning (think water/oil pipelines). I am certain that nobody was asking about this particular site or even the industry its in 15,000 times a minute 24 hours a day. It's much more likely that their crawler is just garbage and got stuck into some kind of loop requesting my domain.

It's common to see them get stuck in a loop on online stores trying every combination of product filter over and over.

Even Googlebot has to be told to not crawl particular querystrings, but the AI crawlers are worse.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#122

Post by p3rls »

PaulDavisThe1st wrote: post #88
kjkjadksj wrote: post #84

Earlier quoted context omitted.

People have been complaining about these crawlers for years well before AI

The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.

i am getting almost 500,000 ai scraper requests a day according to cloudflare's ai audit. google requests the same pages 10+ times each an hour. it was never this bad before.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#123

Post by [deleted] »

vkou wrote: post #76

Why is this not a violation of the CFAA, and why aren't SWEs and directors going to prison over it? As long as I have an EULA or a robots.txt or even a banner that forbids this sort of access, shouldn't any computerized access be considered abuse? Something, something, scraping JSTOR?

[deleted]

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#124

Post by erlend_sh »

PaulDavisThe1st wrote: post #110
tadfisher wrote: post #109

Earlier quoted context omitted.

Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free. [0]: https://dr…

> Tragedy of the commons. No, because there is no such thing, at least not as understood by Garrett Hardin, who put forward the phrase. Commons fail when selfish, greedy people subvert or destroy the governance structures that help control them. If those governance structures exist (and they do for all historical commons) and continue to exist, the commons suffers no tragedy. This recent slide deck talks about Ostrom…

No idea why this is getting downvoted; this is a very important correction since the “tragedy of the commons” meme is based on a flawed premise that needs to be amended.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#125

Post by whatevaa »

renewiltord wrote: post #94
PaulDavisThe1st wrote: post #91

Earlier quoted context omitted.

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ...…

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

That's exactly what they are doing. Rejecting the connection of people like you, cause you don't care. And if you start your own bussiness, you will suddenly encounter the same problem too. Then you will be able to "just write some code".

Anytime somebody writes "just" you immedially can understand that they have no idea what they are talking about.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#126

Post by msgodel »

pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

This isn't really about AI. This is a couple corporations being bad netizens and abusing infrastructure.

The same incentives to do this already existed for search engine operators.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#127

Post by whatevaa »

latexr wrote: post #98
renewiltord wrote: post #95

Earlier quoted context omitted.

Websites aren't people. They don't have desires. Machines have communication protocols. You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. Do what you want with it. People send me spam. I don't whine about it. I block it.

> Websites aren't people. They don't have desires. Obviously I’m talking about the people behind them, and I very much doubt you lack the minimal mental acuity to understand that when I used “website owners” in the preceding sentence. If you don’t want to engage in a good faith discussion you can just say so, no need to waste our time with fake pedantry. But alright, I edited that section. > You can set your machine…

I would suggest not arguing with a wall, the person you are replying to thinks there exists some magic sauce of code to solve this problem.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#128

Post by whatevaa »

mcpar-land wrote: post #53

My worst offender for scraping one of my sites was Anthropic. I deployed an ai tar pit ( https://news.ycombinator.com/item?id=42725147 ) to see what it would do it with it, and Anthropic's crawler kept scraping it for weeks. I calculated the logs and I think I wasted nearly a year of their time in total, because they were crawling in parallel. Other scrapers weren't so persistent.

Tar pits and serve fake but legitimate looking content. Poison it.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#129

Post by msgodel »

renewiltord wrote: post #94
PaulDavisThe1st wrote: post #91

Earlier quoted context omitted.

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ...…

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Sure. What that looks like is always using ssh to access git and things like github going away. I think most of us can agree that's probably not good. For the tools non-technical people use it's probably far worse, pretty much the end of the open web outside static personal pages.

I think the ISPs serving these requests are probably going to have to start going after customers for being abusive in order for this to stop.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#130

Post by BrenBarn »

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publish all those IPs up front?
Post reply on HN