Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

311–320 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#311

Post by reaperducer »

Dylan16807 wrote: post #239
reaperducer wrote: post #187

Earlier quoted context omitted.

robots.txt was invented in 1994(thx chatgpt) Thought of and discussed as a possibility in 1994. Proposed as a standard in 2019. Adopted as a standard in 2022. Thanks, IETF.

This phrasing is very misleading. To bullet point directly from "possibility" to "standard" implies the standardization was a turning point where it could start being used. But it was massively used long before that. The standard is a side note that's barely relevant.

It only became massively used in 2019, when Google recommended it.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#312

Post by slenk »

amy_petrik wrote: post #293
slenk wrote: post #241

Earlier quoted context omitted.

Dude, stop putting words in my mouth. I never said they weren't bad. Some nicer people here tried the educative approach and it worked much better. I learned about Bunny. And I keep forgetting I have a few in deSec but that has a limit. I do not understand the hostility

>How is Cloudflare a parasite? >I never said they weren't bad. >I don't understand the hostility. It's known the community here doesn't like Cloudflare, and anyone who's been on the customer end of Cloudflare would tend to agree. In that context, if you truly are blind to seeing this, when you said, "how is Cloudflare a parasite" to a group not liking of cloudflare... ... it may land as saying something like "How is…

I thought Cloudflare overall was neutral - meaning as many haters as lovers. I know the CEO frequents here as well.

When I ask how is Cloudflare a "parasite" I was being genuine. I know it was a problem for some users, but I don't think I realized how prevalent it was

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#313

Post by Marsymars »

fkyoureadthedoc wrote: post #306
Marsymars wrote: post #151

Earlier quoted context omitted.

Let’s say I’m talking about content that I don’t want behind an auth wall. Is your position simply that all such sites should abandon any efforts to not have the content used for LLM training?

CF will stop bots that respect your robots.txt, and try and stop ones that don't. If your concern is just that you don't want your content used to train an LLM, this will stop the honest companies. If you are concerned about load on your site because the crawlers are hammering your site, the ones that respect robots.txt should be respecting your crawl delay too. CF will be able to block the dumb ones that ignore your…

Sure, I understand all that, but you haven't really answered my question of what an alternative is, you've just laid out why CF is better than nothing. (Or even managing robots.txt manually.)

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#314

Post by fkyoureadthedoc »

Marsymars wrote: post #313
fkyoureadthedoc wrote: post #306

Earlier quoted context omitted.

CF will stop bots that respect your robots.txt, and try and stop ones that don't. If your concern is just that you don't want your content used to train an LLM, this will stop the honest companies. If you are concerned about load on your site because the crawlers are hammering your site, the ones that respect robots.txt should be respecting your crawl delay too. CF will be able to block the dumb ones that ignore your…

Sure, I understand all that, but you haven't really answered my question of what an alternative is, you've just laid out why CF is better than nothing. (Or even managing robots.txt manually.)

Because that depends on your motivations, which I don't know. If you want to prevent your content from being used to train an LLM, CF is not going to prevent that. If you want to protect your site from heavy traffic, CF will do that just like it always has. If you absolutely don't want to your content used to train an LLM, you're basically out of luck if your site is public. So no you shouldn't abandon CF because you get other benefits from it if you need them, but don't expect that you've done anything to prevent your content fro training any given LLM.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#315

Post by Symbiote »

progmetaldev wrote: post #245
Symbiote wrote: post #208

Earlier quoted context omitted.

In theory — in practise I've had to limit Google on two large sites at work. I currently have them limited to 10/s for non-cached requests.

Curious if the content on those sites might have high value to Google? Such as if they have data that is new or unavailable elsewhere, or if they're just standard sites, and you've just been unlucky? I have had odd bot behavior from some major crawlers, but never from Google. I wonder if there is a correlation to usefulness of content, or if certain sites get stuck in a software bug (or some other strange behavior).

Google do value the sites, they have data unavailable elsewhere. At some point we had an automated message saying the site had too many pages and would no longer be indexed, then a human message saying that was a mistake, and our site was an exception to that rule.

But as with any contact with these large companies, our contact eventually disappeared.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#316

Post by MichaelZuo »

Dylan16807 wrote: post #296
MichaelZuo wrote: post #283

Earlier quoted context omitted.

What are you talking about? The prior comment was responding directly to your comment, not any comment preceding that. Of course it’s no longer by definition if you expand the scope beyond an ipv6 whitelist as there are an infinite number of possible whitelists.

> What are you talking about? The first comment with the word "whitelist". Before I entered the conversation. This comment: https://news.ycombinator.com/item?id=44449821 lxgr was challenging the idea that you would treat all IPv6 traffic as suspicious. You justified it by saying that "by definition" "a whitelist system" would do that. I want your definition of "a whitelist system". Not one of the infinite possible de…

None of this even makes sense.

Why does your opinion on how a comment should be interpreted, matter more than anyone else’s opinion in the first place?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#317

Post by jauntywundrkind »

xena wrote: post #275
jauntywundrkind wrote: post #261

Earlier quoted context omitted.

https://xeiaso.net/blog/2025/anubis/ links to https://pod.geraspora.de/posts/17342163 which says: > If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet. My gut is that the switch between IP addresses can't be that hard to follow. That the acc…

This isn't as helpful as you think. If it included all of the HTTP headers that the bots sent and other metadata like TLS ClientHelloInfo it would be a lot more useful.

There's headers, but I hadn't noticed that they are the response headers. :facepalm:

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#318

Post by Dylan16807 »

reaperducer wrote: post #311
Dylan16807 wrote: post #239

Earlier quoted context omitted.

This phrasing is very misleading. To bullet point directly from "possibility" to "standard" implies the standardization was a turning point where it could start being used. But it was massively used long before that. The standard is a side note that's barely relevant.

It only became massively used in 2019, when Google recommended it.

Where did you get that date?

https://serverfault.com/questions/171985/how-can-i-encourage...

Here's a 2010 discussion about Google's explicit support, and I'm sure I could find earlier.

The thing google did in 2019 was submit it as a standard, nothing to do with adoption or starting to recommend. In that very post they said "For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical components of the web" "The proposed REP draft reflects over 20 years of real world experience of relying on robots.txt rules, used both by Googlebot and other major crawlers, as well as about half a billion websites that rely on REP."

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#319

Post by Dylan16807 »

MichaelZuo wrote: post #316
Dylan16807 wrote: post #296

Earlier quoted context omitted.

> What are you talking about? The first comment with the word "whitelist". Before I entered the conversation. This comment: https://news.ycombinator.com/item?id=44449821 lxgr was challenging the idea that you would treat all IPv6 traffic as suspicious. You justified it by saying that "by definition" "a whitelist system" would do that. I want your definition of "a whitelist system". Not one of the infinite possible de…

None of this even makes sense. Why does your opinion on how a comment should be interpreted, matter more than anyone else’s opinion in the first place?

I didn't say that. Huh?

I'm inviting you to tell me how to interpret it. In fact I'm nearly begging you to explain your original comment more. I'm not telling anyone how to interpret it.

I have criticisms for what was said, but that comes after (attempted) interpretation and builds on top of it. I'm not telling anyone how to interpret any post I didn't make.

Edit: In particular, my previous comment has "I assumed" to explain my previous posts, an it has an "If" about what you meant. Neither one of those is telling anyone how to interpret you.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#320

Post by reaperducer »

Dylan16807 wrote: post #318
reaperducer wrote: post #311

Earlier quoted context omitted.

It only became massively used in 2019, when Google recommended it.

Where did you get that date? https://serverfault.com/questions/171985/how-can-i-encourage... Here's a 2010 discussion about Google's explicit support, and I'm sure I could find earlier. The thing google did in 2019 was submit it as a standard, nothing to do with adoption or starting to recommend. In that very post they said "For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical…

Where did you get that date?

On July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under Internet Engineering Task Force.[8]

https://en.wikipedia.org/wiki/Robots.txt

Post reply on HN