Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

91–100 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#91

Post by PaulDavisThe1st »

renewiltord wrote: post #86
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

I don't know if I want your traffic until I see what your traffic is.

You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest!

Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic.

But wait, "your traffic" seems to originate from ... consults fail2ban logs ... more than 900k different IP addresses, so "disconnecting" from you is non-trivial.

I can't put it more politely than this: fuck off. Do not pass go. Do not collect stock options. Go to hell, and stay there.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#92

Post by Fomite »

rnhmjoj wrote: post #54
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

I'd argue it's part of the baked in, fundamental disrespect AI firms have for literally everyone else.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#93

Post by latexr »

renewiltord wrote: post #86
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

> Disconnect from me if you don’t want my traffic.

The problem is precisely that that is not possible. It is very well known that these scrapers aren’t respecting the wishes of website owners and even circumvent blocks any way they can. If these companies respected the website owners’ desires for them to disconnect, we wouldn’t be having this conversation.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#94

Post by renewiltord »

PaulDavisThe1st wrote: post #91
renewiltord wrote: post #86

Earlier quoted context omitted.

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ...…

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#95

Post by renewiltord »

latexr wrote: post #93
renewiltord wrote: post #86

Earlier quoted context omitted.

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

> Disconnect from me if you don’t want my traffic. The problem is precisely that that is not possible. It is very well known that these scrapers aren’t respecting the wishes of website owners and even circumvent blocks any way they can. If these companies respected the website owners’ desires for them to disconnect, we wouldn’t be having this conversation.

Websites aren't people. They don't have desires. Machines have communication protocols. You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. Do what you want with it.

People send me spam. I don't whine about it. I block it.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#96

Post by 1vuio0pswjnm7 »

rnhmjoj wrote: post #54
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

"It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished [sic] conventions (respecting robots.txt, using proper UA string, rate limiting, whatever)."

How does "proper UA string" solve this "blowing up websites" problem

The only thing that matters with respect to the "blowing up websites" problem is rate-limiting, i.e., behaviour

"Shitty crawlers" are a nuisance because of their behaviour, i.e., request rate, not because of whatever UA string they send; the behaviour is what is "shitty" not the UA string. The two are not necessarily correlated and any heuristic that naively assumes so is inviting failure

"Spoofed" UA strings have been facilitated and expected since the earliest web browsers

For example,

https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas...

To borrow the parent's phrasing, the "blowing up websites" problem has nothing to do with UA string specifically

It may have something to do with website operator reluctance to set up rate-limiting though; this despite widespread impelementation of "web APIs" that use rate-limiting

NB. I'm not suggesting rate-limiting is a silver bullet. I'm suggesting that without rate-limiting, UA string as a means of addressing the "blowing up websites" problem is inviting failure

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#97

Post by ronsor »

bgwalter wrote: post #9
8organicbits wrote: post #6

Earlier quoted context omitted.

Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.

The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. I'm not sure how many websites are searched and discarded per query. Since it's the remote, proprietary LLM that initiates the search I would hesitate to call them agents. Maybe "fetcher" is the best term.

> The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load.

So does my browser when I have uBlock Origin enabled.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#98

Post by latexr »

renewiltord wrote: post #95
latexr wrote: post #93

Earlier quoted context omitted.

> Disconnect from me if you don’t want my traffic. The problem is precisely that that is not possible. It is very well known that these scrapers aren’t respecting the wishes of website owners and even circumvent blocks any way they can. If these companies respected the website owners’ desires for them to disconnect, we wouldn’t be having this conversation.

Websites aren't people. They don't have desires. Machines have communication protocols. You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic. Do what you want with it. People send me spam. I don't whine about it. I block it.

> Websites aren't people. They don't have desires.

Obviously I’m talking about the people behind them, and I very much doubt you lack the minimal mental acuity to understand that when I used “website owners” in the preceding sentence. If you don’t want to engage in a good faith discussion you can just say so, no need to waste our time with fake pedantry. But alright, I edited that section.

> You can set your machine to blackhole the traffic or TCP RST or whatever you want. It's just network traffic.

And then you spend all your time in a game of cat and mouse, while these scrappers bring your website down and cost you huge amounts of money. Are you incapable of understanding how that is a problem?

> People send me spam. I don't whine about it. I block it.

Is the amount of spam you get so overwhelming that it swamps your inbox every day to a level you’re unable to find the real messages? Do those spammers routinely circumvent your rules and filters after you’ve blocked them? Is every spam message you get costing you money? Are they increasing every day? No? Then it’s not the same thing at all.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#99

Post by PaulDavisThe1st »

renewiltord wrote: post #94
PaulDavisThe1st wrote: post #91

Earlier quoted context omitted.

I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ...…

There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.

Reject the connection based on what?

IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that.

The URL? Legitimate human (or other) users won't be happy about that.

Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo.

ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#100

Post by ujkhsjkdhf234 »

superkuh wrote: post #67
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

This isn't AI damaging anything. This is corporations damaging things. Same as it ever was. No need for scifi non-human persons when legal corporate persons exist. They latch on to whatever big new thing in tech that people don't understand which comes along and brand themselves with it and cause damage trying to make money; even if they mostly fail at it. And for most actual humans they only ever see or interact wit…

Cloudflare exists because people can't be good stewards of the internet.

> This isn't AI damaging anything. This is corporations damaging things

This is the guns don't kill people, people kill people argument. The problem with crawlers is about 10x worse than it was previously because of AI and their hunger for data.

Post reply on HN