Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

191–200 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#191

Post by lofaszvanitt »

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

Cyberpunk aged well. "You better not be on the unprotected internet". Too many hazards out there. Rogue AIs and other shit...

Cloudflare is here to protecc you from all those evils. Just come under our umbrella.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#192

Post by lxgr »

friedtofu wrote: post #186
tcdent wrote: post #174

Earlier quoted context omitted.

[flagged]

What the hell... Even if you're directing this at the user's blog posts specifically; this is a ridiculously pessimistic, sad way to view things. I hope you're just having a bad day because if you sincerely have this greedy, cynical mindset day to day(towards blogging, software, offline/real life activities, whatever) I feel sorry for you.

GP's comment might be provocatively phrased, but I don't think it's an invalid point to have:

When I publish something online for free, i.e. without requiring authentication or payment, be it a Reddit comment, a blog post, a Stackoverflow answer or anything else, I do so hoping that it will be useful to somebody somehow, without any illusions about being able to gatekeep some types of current or future consumers.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#194

Post by lxgr »

jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

That's at least a more reasonable default than that I've seen at least one newspaper do, which is to block both LLM scrapers and things like ChatGPT's search feature explicitly.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#195

Post by lxgr »

postalcoder wrote: post #28
jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This…

More and more people use ChatGPT for search, so blocking that doesn't seem like a successful strategy long-term.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#196

Post by lxgr »

MichaelZuo wrote: post #119
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

If your on ipv6, I think they have to for ipv6 addresses… there’s just way too many bots and way too many addresses to feasibly do anything more precise. If your on ipv4 you should check whether your behind a NAT otherwise you may have gotten an address that was previously used by a bot network.

> I think they have to for ipv6 addresses… there’s just way too many bots and way too many addresses

Are you really arguing that it's legitimate to consider all IPv6 browsing traffic "suspicious"?

If anything, I'd say that IPv4 is probably harder, given that NATs can hide hundreds or thousands of users behind a single IPv4 address, some of which might be malicious.

> you may have gotten an address that was previously used by a bot network.

Great, another "credit score" to worry about...

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#197

Post by sealeck »

lxgr wrote: post #185
slenk wrote: post #120

Earlier quoted context omitted.

How is Cloudflare a parasite? I can use Cloudflare, and get their AI protection, for free. I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime.

> I have dozens of domains I have used with Cloudflare at one point and I haven't paid them a dime. Maybe you haven't, but your users (primarily those using "suspicious" operating systems and browsers) certainly have – with their time spent solving captchas.

But Cloudflare have removed CAPTCHAs

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#198

Post by bawolff »

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

I think its 100% ok to freely train on public internet data.

What is absolutely not ok is to crawl at such an excessive speed that it makes it difficult to host small scale websites.

Truly a tragedy of the commons.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#199

Post by axus »

andy99 wrote: post #112
abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

From the server perspective Cloudflare is solving problems and not causing problems to other servers.

Analogy: locks for high-value items in grocery stores are annoying to customers, but other stores aren't being coerced by the locksmith to use them.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#200

Post by lxgr »

godelski wrote: post #127
jefftk wrote: post #110

Earlier quoted context omitted.

I write online (comments here, open source software, blogging, etc) because I have ideas I want to share. Whether it's "I did a thing and here's how" or "we should change policy in this specific way" or "does anyone know how to X" I'm happy for this to go into training models just like I'm happy for it to go into humans reading.

Tbh, that content I'm mostly fine with. My only real issue is that people are making trillions off the free labor of people like you and me, giving less time to create that OSS and blogs. But this isn't new to AI, it is just scaled. What I do care about is the theft of my identity. A person may learn from the words I write but that person doesn't end up mimicking the way I write. They are still uniquely themselves. I…

> What I do care about is the theft of my identity. A person may learn from the words I write but that person doesn't end up mimicking the way I write. They are still uniquely themselves.

Of course they do, to some extent. Just because it's been infeasible to track the exact "graph of influence", that's literally how humans have learned to speak and write for as long as we've had language and writing.

> I'm concerned that the more I write the more my text becomes my identifier. I use a handle so I can talk more openly about some issues.

That's a much more serious concern, in my view. But I believe that LLMs are both the problem and solution here: "Remove style entropy" is just a prompt away, these days.

Post reply on HN