Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

271–280 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#271

Post by brendyn »

slenk wrote: post #120
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

A parasite leaches off it's host to the hosts harm. Maybe it's not a good analogy, but Im in china, and it's painful after paying money for a VPN to bypass censorship to find myself routinely blocked by CDNs because they decided I'm not human. I'm honestly feeling more opressed by these middlemen than the government sometimes. For example, maybe I can't log in to a game due to being blocked by the login API, and the game company just responds by telling me to run an antivirus scanner and try again since they are not personally developing that system that lack awareness. Such people with genuine need for VPNs and privacy tools are the sacrifice for this system.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#272

Post by Dylan16807 »

MichaelZuo wrote: post #257
Dylan16807 wrote: post #237

Earlier quoted context omitted.

> For a whitelist system, then by definition yes? A whitelist system would consider all IPv4 traffic suspicious by default too. This is not an answer to why you'd be suspicious of IPv6 in particular. > I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses. Handling /56s or something like that is about the same as handling individual IPv4 addresses.

> A whitelist system would consider all IPv4 traffic suspicious by default too. Based on what argument…?

The definition of whitelisting. The argument you brought up.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#273

Post by BeetleB »

jefftk wrote: post #229
BeetleB wrote: post #206

Earlier quoted context omitted.

> A person may learn from the words I write but that person doesn't end up mimicking the way I write. Oh, I wish I could get AI to mimic the way I write! I'd pay money for it. I often want to type up an email/doc/whatever but don't because of occasional RSI issues. If I could get an AI to type it up for me while still sounding like me - that would be a big boon for my health.

> but don't because of occasional RSI issues I also have this issues that often keep me from typing, but FYI dictation has gotten very good. (Dictated this)

Oh yeah, I use dictation and then clean it up with GPT. It's awesome. But I speak very differently from how I write. So I'd like to dictate it, and then have it rewrite it in my writing style.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#274

Post by MichaelZuo »

Dylan16807 wrote: post #272
MichaelZuo wrote: post #257

Earlier quoted context omitted.

> A whitelist system would consider all IPv4 traffic suspicious by default too. Based on what argument…?

The definition of whitelisting. The argument you brought up.

No…? Someone can clearly implement a whitelist system that applies only to ipv6… but that makes no judgement on ipv4.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#275

Post by xena »

jauntywundrkind wrote: post #261
jowea wrote: post #210

Earlier quoted context omitted.

I'm not an expert on website hosting, but after reading some of the blog posts on Anubis, those people were truly at wit's end trying to block AI scrappers with techniques like the ones you imply.

https://xeiaso.net/blog/2025/anubis/ links to https://pod.geraspora.de/posts/17342163 which says: > If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet. My gut is that the switch between IP addresses can't be that hard to follow. That the acc…

This isn't as helpful as you think. If it included all of the HTTP headers that the bots sent and other metadata like TLS ClientHelloInfo it would be a lot more useful.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#276

Post by mattl »

xena wrote: post #254
mattl wrote: post #246

Earlier quoted context omitted.

The proof of work stuff feels so cryptocurrency adjacent that I've been looking at other tools for my own thing, but I've seen Anubis on other websites and it seems to do a good job.

There's a non proof of work challenge: https://anubis.techaro.lol/docs/admin/configuration/challeng... Also: Anubis does not mine cryptocurrency. Proof of work is easy to validate on the server and economically scales poorly in the wild for abusive scrapers.

Thanks for the link. I’ll have a look.

I’m glad there’s no cryptocurrency involved (was never a concern) but I worry about the optics of something so closely associated.

(I appreciate your commenting on this. I know the project recently blew up in popularity. Keep up the great work)

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#277

Post by Dylan16807 »

MichaelZuo wrote: post #274
Dylan16807 wrote: post #272

Earlier quoted context omitted.

The definition of whitelisting. The argument you brought up.

No…? Someone can clearly implement a whitelist system that applies only to ipv6… but that makes no judgement on ipv4.

Let's back up a step. You said by definition a whitelist system would consider every IPv6 suspicious (until it's put on the list, presumably). What is that definition?

If "applies only to IPv6" is an optional decision someone could make, then it's not part of the definition of a whitelist system for IPs, right?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#279

Post by lxgr »

sealeck wrote: post #216
lxgr wrote: post #201

Earlier quoted context omitted.

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

https://blog.cloudflare.com/end-cloudflare-captcha/

It's not much consolation to me if I'm one of the 25% still being challenged.

The world really has more than enough heuristic fraud detection systems that most people aren't even aware exist, but make life miserable for those that somehow don't fit the typical user profile. In fact, the lower the false positive rate gets, the more painful these systems usually become for each false positive.

I'm so tired of it. Sure, use ML (or "AI") to classify good and evil users for initial triaging all day long, but make sure you have a non-painful fallback.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#280

Post by DanOpcode »

Symbiote wrote: post #203
lxgr wrote: post #201

Earlier quoted context omitted.

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

Since a few days ago, I've been getting Captchas hourly or more. It's probably because I use Firefox on Linux with an ad blocker. For my part, I've ensured we don't use Cloudflare at work.

It must depend on something else. Firefox & Linux have always worked fine for me, I cannot remember when I last got restricted by a Cloudflare captcha.
Post reply on HN