Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

241–250 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#241

Post by slenk »

djfivyvusn wrote: post #233
slenk wrote: post #227

Earlier quoted context omitted.

Where did I say all. Most of the time I don't use them for their network, usually just DNS records for mail because their interface is nicer than namecheap and gives me basic stats. To my understanding, they aren't blocking MX records behind captchas

So you're not using the parasite and that's your claim why it's not a parasite?

Dude, stop putting words in my mouth. I never said they weren't bad.

Some nicer people here tried the educative approach and it worked much better. I learned about Bunny. And I keep forgetting I have a few in deSec but that has a limit.

I do not understand the hostility

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#242

Post by shlomo_z »

slenk wrote: post #120
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Did you read his comment? He explained the issue he has with Cloudflare...

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#243

Post by xena »

mattl wrote: post #159
dceddia wrote: post #135

Earlier quoted context omitted.

Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature. If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick? The thing about this slow slide is it’s always defensible. Someone ca…

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

I make this: https://anubis.techaro.lol. I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#244

Post by slenk »

aorth wrote: post #164
slenk wrote: post #147

Earlier quoted context omitted.

Right - do I want them getting some info from me, or do I want my IP address exposed? Besides CloudFront, which still costs money, what other option is there for semi-privacy and caching for free?

As the old addage goes: If you're not paying for it, you're the product. Lots of nuance, but generally: pay for things you use. Servers, engineers, and research and development are not free, so someone has to pay.

fwiw, I have been convinced to look at other options

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#245

Post by progmetaldev »

Symbiote wrote: post #208
mrweasel wrote: post #88

Earlier quoted context omitted.

Very few people seems to be complaining that Google crashes their sites. Google also publish their crawlers IP ranges, but you really don't need to rate-limit Google, they know how to back off and not overload sites.

In theory — in practise I've had to limit Google on two large sites at work. I currently have them limited to 10/s for non-cached requests.

Curious if the content on those sites might have high value to Google? Such as if they have data that is new or unavailable elsewhere, or if they're just standard sites, and you've just been unlucky?

I have had odd bot behavior from some major crawlers, but never from Google. I wonder if there is a correlation to usefulness of content, or if certain sites get stuck in a software bug (or some other strange behavior).

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#246

Post by mattl »

xena wrote: post #243
mattl wrote: post #159

Earlier quoted context omitted.

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

I make this: https://anubis.techaro.lol . I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.

The proof of work stuff feels so cryptocurrency adjacent that I've been looking at other tools for my own thing, but I've seen Anubis on other websites and it seems to do a good job.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#247

Post by mattl »

1oooqooq wrote: post #221
mattl wrote: post #159

Earlier quoted context omitted.

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

it's called not having a vibecoded app that falls to pieces on public endpoints even before ngix ratelimit can kick in

Nobody is talking about a vibe coded app. I want to block AI scrapers entirely.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#248

Post by mattl »

saint_yossarian wrote: post #211
mattl wrote: post #159

Earlier quoted context omitted.

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

I remember using mod_security with Apache long ago for some of this, looks like it's still around and now also supports Nginx and IIS: https://modsecurity.org/

Thank you. This doesn't have everything I'm looking for, but apparently it has been packaged in Debian at least. I don't know why the website doesn't mention this.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#249

Post by sitzkrieg »

lxgr wrote: post #201
sealeck wrote: post #197

Earlier quoted context omitted.

But Cloudflare have removed CAPTCHAs

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

my residental ip of years (which is not shared or cgnat) was recently flagged by cloudflare for who knows why. if you are asking, you havent seen when cloudflare thinks you are something else.

cloudflare are not the good guys because they give people free cdn and ddos protection lol

Post reply on HN