Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

141–148 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#141

Post by rco8786 »

dwd wrote: post #121
rco8786 wrote: post #50

Earlier quoted context omitted.

The site is about a particular type of pipeline cleaning (think water/oil pipelines). I am certain that nobody was asking about this particular site or even the industry its in 15,000 times a minute 24 hours a day. It's much more likely that their crawler is just garbage and got stuck into some kind of loop requesting my domain.

It's common to see them get stuck in a loop on online stores trying every combination of product filter over and over. Even Googlebot has to be told to not crawl particular querystrings, but the AI crawlers are worse.

Not an online store just a bog standard Wordpress site

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#142

Post by snowwrestler »

bgwalter wrote: post #30
snowwrestler wrote: post #12

Earlier quoted context omitted.

While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses. The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached. Corpus-building craw…

Why would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the search function, as mentioned in the top comments.

It says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers.

Of course they are crawling every day to improve their training data. The goal is LLMs that know everything, but “everything” changes on a daily basis.

Meta and OpenAI are simply the largest after Google, but Google has had ~20 more years to learn how to politely operate crawlers at full-Internet scale.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#143

Post by HankStallone »

masfuerte wrote: post #70
okasaki wrote: post #52

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either coopera…

A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots. The bot operators DNGAF.

Right. I don't care if AI (or anything else) indexes or learns from my sites. That's what they're there for. But yesterday I blocked an IP that hit one of my sites 82000 times in an hour, or 22/second. And apparently it's a very stupid bot, because it kept redownloading CSS and other asset files every time it saw a link to them.

There's no way the people behind that bot are going to follow any suggestions to make it behave better. After all, adding things like caching and rate-limiting to your web crawler might take a few hours, and who's got time for that.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#144

Post by PaulDavisThe1st »

Nextgrid wrote: post #117
ben_w wrote: post #101

Earlier quoted context omitted.

Reject 900k different connections from different origins each asking for what would in isolation be fine and the only problem is the quantity?

But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of t…

There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison.

These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK.

Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't.

And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address.

(*) put differently, if all this current scraping activity delivered some notable benefit to humanity

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#145

Post by 1vuio0pswjnm7 »

AbortedLaunch wrote: post #102
1vuio0pswjnm7 wrote: post #96

Earlier quoted context omitted.

"It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished [sic] conventions (respecting robots.txt, using proper UA string, rate limiting, whatever)." How does "proper UA string" solve this "blowing up websites" problem The only thing that matters with respect to the "blowing up websites" problem is rate-limiting, i.e., behaviour "Shitty crawlers" are a nui…

Some of these crawlers appear to be designed to avoid rate limiting based on IP. I regularly see millions of unique ips doing strange requests, each just one or at most a few per day. When a response contains a unique redirect I often see a geographically distinct address fetching the destination.

"Some of these crawlers appear to be designed to avoid rate limiting based on IP."

Unless the rate is exceeded, the limit is not being avoided

"I regularly see millions of unique ips doing strange requests, each just one or at most a few per day."

Assuming the rate limit is more than one or a few requests every 24h this would be complying with the limit, not avoiding it

It could be that sometimes the problem website operators are concerned about is not "website load", i.e., the problem the article is discussing, it is actually something else (NB. I am not speculating about this particular operator, I am making a general observation)

If a website is able to fulfill all requests from unique IPs without affecting quality of service, then it stands to reason "website load" is not a problem the website operator is having

For example, the article's title claims Meta is amongst the "worst offenders" of creating excessive website load caused by "AI crawlers, fetchers"

Meta has been shown to have used third party proxy services wth rotating IP addresses in order to scrape other websites; it also sued one of these services because it was being used to scrape Meta's website, Facebook

https://brightdata.com/blog/general/meta-dismisses-claim-aga...

Whether the problem that Meta was having with this "scraping" was "website load" is debatable; if the requests were being fulfilled without affecting QoS, then arguably "website load" was not a problem

Rate-limiting addresses the problem of website load; it allows website operators to ensure that requests from all IP addresses are adequately served as opposed to preferentially servicing some IP addresses to the detriment of others (degraded QoS)

Perhaps some website operators become concerned that many unique IP addresses may be under the control of a single entity, and that this entity may be a competitor; this could be a problem for them

But if their website is able to fulfill all the requests it receives without degrading QoS then arguably "website load" is not a problem they are having

NB. I am not suggesting that a high volume of requests from a single entity, each complying with a rate-limit is acceptable, nor am I making any comment about the practice of "scraping" for commercial gain. I am only commenting about what rate-limiting is designed to do and whether it works for that purpose

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#146

Post by renewiltord »

pjc50 wrote: post #114
renewiltord wrote: post #86

Earlier quoted context omitted.

If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.

People are doing exactly that. And then other people who want to use the website are asking why they get blocked by false positives.

Yeah, it just seems like things are playing out as one would expect them to. You're right

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#147

Post by weaksauce »

bwb wrote: post #85
weaksauce wrote: post #64

Earlier quoted context omitted.

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

Got a good link to something on github that does this? I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.

here's one example using rack attack on a rails app:

https://github.com/pinballmap/pbm/blob/302ac638850711878ac61...

https://github.com/pinballmap/pbm/blob/302ac638850711878ac61...

but it only bans for 3 hours. if they don't respect the hidden link and robots.txt they get banned.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#148

Post by tbonnin »

BrenBarn wrote: post #130

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publ…

Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t. Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs outside of their declared list [3].

[1] https://platform.openai.com/docs/bots

[2] https://docs.mistral.ai/robots/

[3] https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Post reply on HN