Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

221–230 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#221

Post by 1oooqooq »

mattl wrote: post #159
dceddia wrote: post #135

Earlier quoted context omitted.

Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature. If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick? The thing about this slow slide is it’s always defensible. Someone ca…

We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it. * curl | bash * Docker * Anything that smacks of cryptocurrency or other scams Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it. (apt/dnf install no-bad-actors) Until that exists, using Cloudflare…

it's called not having a vibecoded app that falls to pieces on public endpoints even before ngix ratelimit can kick in

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#222

Post by Sebguer »

sealeck wrote: post #216
lxgr wrote: post #201

Earlier quoted context omitted.

Not sure if you're joking, but if you're not: Congratulations on using a very "normal/safe" OS/browser/IP. I get captchas daily, without using any VPN and on several different IPs (work, home, mobile). The only crime I can think of is that I'm using Firefox instead of Chrome.

https://blog.cloudflare.com/end-cloudflare-captcha/

Managed challenges are just CAPTCHA by another name.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#223

Post by slenk »

bombcar wrote: post #217
slenk wrote: post #120

Earlier quoted context omitted.

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Download Brave. Turn on Tor and browse for a week. Now you know what “undesirables” feel like, where “undesirables” can be from a poor country, a bad IP block, outdated browsers, etc. It sucks.

I already said in another post I am looking at Bunny, but they also don't seem to want to take my money. I've tried 3 cards. I am willing to pay for a good service, but I will be honest, I don't know many of cloudflare's competitors

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#224

Post by t1001 »

sneak wrote: post #188

This idea that you can publish data for people to download and read but not for people to download and store, or print, or think about, or train on is a doomed one. If you don’t want people reading your data, don’t put it on the web. The concept that copyright extends to “human eyeballs only” is a silly one.

With the problem being bots hammering the site en masse, it feels like the better analog is "allowing free replicator use without having someone ruin the fun by requesting ten tons of food be produced in their quarters every minute".

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#225

Post by chaoskitty »

slenk wrote: post #120
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Serious question: You put Cloudflare between all your domains and all your visitors without looking in to how this would affect your site's reachability? If so, that's interesting, considering that many people in this community are negatively affected by Cloudflare because they're using Linux and/or some less than mainstream browser.

You might want to read some threads on here about Cloudflare.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#226

Post by pixl97 »

sneak wrote: post #190
Bender wrote: post #115

Earlier quoted context omitted.

For my silly hobby sites I just return status 444 close the connection for anything that has case-insentive "bot" in the UA requesting anything other than robots.txt, humans.txt, favicon.ico, etc... This would also drop search engines but I blackhole route most of their CIDR blocks. I'm probably the only one here that would do this.

How does a bot scraping your silly hobby sites for any purpose harm or negatively affect you in any way?

Depends if they hit a site enough to make it cost something. It's not hard for bots to flood servers.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#227

Post by slenk »

chaoskitty wrote: post #225
slenk wrote: post #120

Earlier quoted context omitted.

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

Serious question: You put Cloudflare between all your domains and all your visitors without looking in to how this would affect your site's reachability? If so, that's interesting, considering that many people in this community are negatively affected by Cloudflare because they're using Linux and/or some less than mainstream browser. You might want to read some threads on here about Cloudflare.

Where did I say all.

Most of the time I don't use them for their network, usually just DNS records for mail because their interface is nicer than namecheap and gives me basic stats.

To my understanding, they aren't blocking MX records behind captchas

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#228

Post by progmetaldev »

ijk wrote: post #70
fennecfoxy wrote: post #31

Earlier quoted context omitted.

With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Goo…

What I want to know is if the flood of scraping everyone has been complaining about is coming from people trying to scrape for training or bots doing RAG search. I get that everyone wants data, but presumably the big players already scraped the web. Do they really need to do it again? Or is it bit players reproducing data that's likely already in the training set? Or is it really that valuable to have your own scrape…

I believe it's both. We're at a place where legislation hasn't really declared what is and isn't allowed. These scrapers are acting like Googlebot or any other search engine crawler, and trying to find any kind of new content that might be of value to their users.

New data is still being added online daily (probably hourly, if not more often) by humans, and the first ones to gain access could be the "winners," particularly if their users happen to need up to date data (and the service happens to have scraped it). Just like with search engines/crawlers, there's also the big players that may respect your website, but there are also those that don't use rate-limiting or respect robots.txt.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#229

Post by jefftk »

BeetleB wrote: post #206
godelski wrote: post #127

Earlier quoted context omitted.

Tbh, that content I'm mostly fine with. My only real issue is that people are making trillions off the free labor of people like you and me, giving less time to create that OSS and blogs. But this isn't new to AI, it is just scaled. What I do care about is the theft of my identity. A person may learn from the words I write but that person doesn't end up mimicking the way I write. They are still uniquely themselves. I…

> A person may learn from the words I write but that person doesn't end up mimicking the way I write. Oh, I wish I could get AI to mimic the way I write! I'd pay money for it. I often want to type up an email/doc/whatever but don't because of occasional RSI issues. If I could get an AI to type it up for me while still sounding like me - that would be a big boon for my health.

> but don't because of occasional RSI issues

I also have this issues that often keep me from typing, but FYI dictation has gotten very good.

(Dictated this)

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#230

Post by jefftk »

blibble wrote: post #166
bob1029 wrote: post #154

Earlier quoted context omitted.

> OSS > people are making trillions off the free labor of people like you and me I read "No Discrimination Against Fields of Endeavor" to also include LLMs and especially the cases that we most deeply disagree with. Either we believe in the principles of OSS or we do not. If you do not like the idea of your intellectual property being used for commercial purposes then this model is definitely not for you. There is no…

> Either we believe in the principles of OSS or we do not. If you do not like the idea of your intellectual property being used for commercial purposes then this model is definitely not for you. I've been writing open source for more than 20 years I gave away my work for free with one condition: leave my name on it (MIT license) the AI parasites then strip the attribution out they are the ones violating the principle…

> I suggest every developer does the same until the legal position is clarified (in our favour)

There are a lot of people developing open source software with a wide range of goals. In my case, I'm totally happy for LLMs to learn from my coding, just like they've learned from millions of other peoples. I wouldn't want them to duplicate it verbatim, but (due to copyright filters + that not usually being the best way to solve a problem) they don't.

Post reply on HN