Cloudflare Introduces Default Blocking of A.I. Data Scrapers • Hacker News Board

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#261

Post by jauntywundrkind » Thu, Jul 03, 2025, 1:59 AM UTC

jowea wrote: post #210

jauntywundrkind wrote: post #113

Earlier quoted context omitted.

I too am a bit confused / mystified at the strong reaction. But I do expect a lot of badly optimized sites that just want out. I struggle to think of a web related library that has spread faster than Anubis checker. It's everywhere now! https://github.com/TecharoHQ/anubis I'm surprised we don't see more efforts to rate limit. I assume many of these are distributed crawlers, but it feels like there's got to be pools o…

I'm not an expert on website hosting, but after reading some of the blog posts on Anubis, those people were truly at wit's end trying to block AI scrappers with techniques like the ones you imply.

https://xeiaso.net/blog/2025/anubis/ links to https://pod.geraspora.de/posts/17342163 which says:

> If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

My gut is that the switch between IP addresses can't be that hard to follow. That the access pattern it pretty obvious to follow across identities.

But it would be non trivial, it would entail crafting new systems and doing new work per request (when traffic starts to be elevated, as a first gate).

Just making the client run through some math gauntlet is an obvious win that aggressors probably can't break. But I still think there's probably some really good hanging fruit for identifying and rate limiting even these somewhat rather more annoying traffic patterns, that the behavior itself leaves a figure print that can't be hidden and which can absolutely be rate limited. And I'd like to see that area explored.

Edit: oh heck yes, new submission with 1.7tb logs of what AI crawlers do. Now we can machine learn some better rate limiting techniques! https://news.ycombinator.com/item?id=44450352 https://huggingface.co/datasets/lee101/webfiddle-internet-ra...

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#262

Post by trollbridge » Thu, Jul 03, 2025, 1:59 AM UTC

arkmm wrote: post #156

dolebirchwood wrote: post #123

Earlier quoted context omitted.

Thank you for having this attitude. I have never attempted any blogging because I always figured no one is actually going to read it. With LLMs, however, I know they will. I actually see this as a motivation to blog, as we are in a position to shape this emerging knowledge base. I don't find it discouraging that others may be profiting off our freely published work, just as I myself have benefited tremendously from o…

This is an interesting take, thanks for sharing. I wonder how someone should adjust their blogging if they believe their primary audience will be LLMs.

There’s a few instances of things I stated (about historical topics or very narrow topics in sociology) that were incorrect. LLMs scraped these off of web forums or other places, and now these bogus “facts” are permanently embedded into LLM models, because nobody else ever really talked about the specific topic.

Most amusingly, someone cited LLM generated output about this telling me how this “fact” is true when I was telling them it’s not true.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#263

Post by trollbridge » Thu, Jul 03, 2025, 2:01 AM UTC

godelski wrote: post #124

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

Including your comment, including this comment. HN itself is routinely scraped. What makes me most uncomfortable is deanonymization via speech analysis. It's something we can already do but is hard to do at scale. This is the ultimate tool for authoritarians. There's no hidden identities because your speech is your identifier. It is without borders. It doesn't matter if your government is good, a bad acting governmen…

The degree to which people say “self-delete” and “unalive” is absurd these days and I now hear it in real life.

It’s Orwellian in the truest sense of the word.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#264

Post by qskousen » Thu, Jul 03, 2025, 2:03 AM UTC

visarga wrote: post #252

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

> everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop I think on the contrary, who sets the prompts stands to get benefits, the AI provider gets a flat fee, and authors get nothing except the same AI tools as anyone else. That is natural since the users are bringing the problem to the AI, of cou…

That's a really interesting way to think about it, thank you! I've always had a kind of "gut feeling" that AI training on our data is fine with me, but without really thinking too much about why. I think this explains what I've been feeling.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#265

Post by SchemaLoad » Thu, Jul 03, 2025, 2:11 AM UTC

lxgr wrote: post #201

sealeck wrote: post #197

Earlier quoted context omitted.

But Cloudflare have removed CAPTCHAs

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

I use a VPN and firefox and I get some extra captchas but not enough to be annoying. And you don't have to do anything more than tap the checkbox.

Meanwhile a bunch of "security" products other websites use just flat out block you if you're on a VPN. Other sites like youtube or reddit are in between where they block you unless you are logged in.

Cloudflare is the least obtrusive of the options.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#266

Post by SchemaLoad » Thu, Jul 03, 2025, 2:12 AM UTC

bombcar wrote: post #217

slenk wrote: post #120

Earlier quoted context omitted.

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Download Brave. Turn on Tor and browse for a week. Now you know what “undesirables” feel like, where “undesirables” can be from a poor country, a bad IP block, outdated browsers, etc. It sucks.

It's kind of an impossible problem though. They either save some tracking cookie to link your sessions between websites, or they have to re captcha check you on every website.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#267

Post by SchemaLoad » Thu, Jul 03, 2025, 2:14 AM UTC

trollbridge wrote: post #259

rockskon wrote: post #134

Earlier quoted context omitted.

LLM scrapers have dramatically been increasing the cost of hosting various small websites. Without something being done, the data that these scrapers rely on would eventually no longer exist.

I use Cloudflare and edge caching, so it doesn’t really affect me, but the amount of LLM scraping of various static assets for apps I host is ridiculous. We’re talking a JavaScript file of strings to respond like “login failed”, “reset your password” just over and over again. Hundreds of fetches a day, often from what appears to be the same system.

Turn on the the Cloudflare tarpit. When it detects LLM scrapers it starts generating infinite AI slop pages to feed the scrapers. Ruining their dataset and keeping them off your actual site.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#268

Post by joshdavham » Thu, Jul 03, 2025, 2:15 AM UTC

cmg wrote: post #2

Archive link: https://archive.ph/ARnyu

How did you make that link?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#269

Post by SchemaLoad » Thu, Jul 03, 2025, 2:15 AM UTC

bawolff wrote: post #198

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

I think its 100% ok to freely train on public internet data. What is absolutely not ok is to crawl at such an excessive speed that it makes it difficult to host small scale websites. Truly a tragedy of the commons.

This is the attitude that's going to kill the public internet. Because you're right, it is a free for all right now with the only way to opt out being putting content behind restricted platforms.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#270

Post by o11c » Thu, Jul 03, 2025, 2:22 AM UTC

robrenaud wrote: post #251

o11c wrote: post #145

Earlier quoted context omitted.

It's not philosophical, it's economical. AI scrapers increase traffic by maybe 10x (this varies per site) but provide no real value whatsoever to anyone . If you look at various forms of "value": * Saying "this uses AI" might make numbers go up on the stock market if you manage to persuade people it will make numbers go up (see also: the market will remain irrational longer than you can remain solvent). * Saying "thi…

Why are 100s of millions of people using AI if it is providing no value?

Because it's injected into a previously working product - even if it makes it worse - and automatically injects its ideas. That counts as "somebody using it".

Because it's bundled with other products that do provide value, and that counts as "someone using it".

Because some middle manager has declared that I must add AI to my workflow ... somehow. Whatever, if they want to pay me to accomplish less than usual, that's not my problem.

Because it's a cool new toy to play around with a bit.

Because surely all these people saying "AI is useful now" aren't just lying shills, so we'd better investigate their claims again ... nope, still terminally broken.