Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

211–220 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#211

Post by saint_yossarian »

mattl wrote: post #159
dceddia wrote: post #135

Earlier quoted context omitted.

Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature. If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick? The thing about this slow slide is it’s always defensible. Someone ca…

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

I remember using mod_security with Apache long ago for some of this, looks like it's still around and now also supports Nginx and IIS: https://modsecurity.org/

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#212

Post by jowea »

TechDebtDevin wrote: post #53
dawnerd wrote: post #47

I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically. It also completely put a stop to perplexity as far as I can tell. And the rob…

This does nothing dude. Literally nothing. OpenAI or whoever are just going to hire people like me who dont get caught. Stop ruining the experience of users and allowing cf to fill the internet with more bloated javascript challenge pages and privacy invading fingerprinting. Stop making cf the police of the internet. We're literally handing the internet to this company on a silver platter to do MITM attacks on our pr…

You want them to pay the server costs to serve content to AI scrappers for free? The alternative is Anubis, which is maybe equally annoying to users in a different way.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#213

Post by Symbiote »

Meekro wrote: post #32

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…

That's a pretty big assumption.

The largest site I work on has 100,000s of pages, each in around 10 languages — that's already millions of pages.

It generally works fine. Yesterday it served just under 1000 RPS over the day.

AI crawlers have brought it down when a single crawler has added 100, 200 or more RPS distributed over a wide range of IPs — it's not so much the number of extra requests, though it's very disproportionate for one "user", but they can end up hitting an expensive endpoint excluded by robots.txt and protected by other rate-limiting measures, which didn't anticipate a DDoS.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#214

Post by MichaelZuo »

lxgr wrote: post #196
MichaelZuo wrote: post #119

Earlier quoted context omitted.

If your on ipv6, I think they have to for ipv6 addresses… there’s just way too many bots and way too many addresses to feasibly do anything more precise. If your on ipv4 you should check whether your behind a NAT otherwise you may have gotten an address that was previously used by a bot network.

> I think they have to for ipv6 addresses… there’s just way too many bots and way too many addresses Are you really arguing that it's legitimate to consider all IPv6 browsing traffic "suspicious"? If anything, I'd say that IPv4 is probably harder, given that NATs can hide hundreds or thousands of users behind a single IPv4 address, some of which might be malicious. > you may have gotten an address that was previously…

For a whitelist system, then by definition yes?

If it’s a blacklist system, like I said I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#215

Post by godelski »

lxgr wrote: post #189
godelski wrote: post #125

Earlier quoted context omitted.

I'm in a pretty similar boat except I frequently hit challenges. Especially if I use a VPN (which is more trustworthy than my ISP). Ironically, I'm using Cloudflare for DoH

I'd be surprised if Cloudflare were actually correlating DoH requests to HTTP requests following them, so I don't think that's a signal they are likely to use.

Probably not. In fact, it's probably a good sign that they are being accurate about that traffic being encrypted.

But I did find it ironic

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#216

Post by sealeck »

lxgr wrote: post #201
sealeck wrote: post #197

Earlier quoted context omitted.

But Cloudflare have removed CAPTCHAs

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

https://blog.cloudflare.com/end-cloudflare-captcha/

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#217

Post by bombcar »

slenk wrote: post #120
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Download Brave.

Turn on Tor and browse for a week.

Now you know what “undesirables” feel like, where “undesirables” can be from a poor country, a bad IP block, outdated browsers, etc.

It sucks.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#218

Post by 1oooqooq »

sealeck wrote: post #197
lxgr wrote: post #185

Earlier quoted context omitted.

> I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime. Maybe you haven't, but your users (primarily those using "suspicious" operating systems and browsers) certainly have – with their time spent solving captchas.

But Cloudflare have removed CAPTCHAs

you forgot /s

(the people not getting the joke, yes the new system don't make you train any image recognition dataset, but they profile the hell out anything they can get their hand on just like google captcha and then if you smell like a bot you're denied access. goodbye)

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#219

Post by 1oooqooq »

fsflover wrote: post #133
slenk wrote: post #120

Earlier quoted context omitted.

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

They put themselves as a middle man for almost the whole Internet, collect huge usage data about everyone and block anybody who doesn't use mainstream tools: https://news.ycombinator.com/item?id=42953508 https://news.ycombinator.com/item?id=13718752 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=41864632 https://news.ycombinator.com/item?id=42577076

https://infosec.exchange/@k3ym0/114762301792775770

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#220

Post by 1oooqooq »

AnthonyMouse wrote: post #182
fsflover wrote: post #133

Earlier quoted context omitted.

They put themselves as a middle man for almost the whole Internet, collect huge usage data about everyone and block anybody who doesn't use mainstream tools: https://news.ycombinator.com/item?id=42953508 https://news.ycombinator.com/item?id=13718752 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=41864632 https://news.ycombinator.com/item?id=42577076

You can add another one as a result of this article: The data you need to train AI and the data you need to build a search engine are the same data. So now they're inhibiting every new search engine that wants to compete with Google.

they always had. this post is about turning the false positives "up to 11" with impunity
Post reply on HN