Cloudflare Introduces Default Blocking of A.I. Data Scrapers • Hacker News Board

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#21

Post by xyst » Wed, Jul 02, 2025, 2:11 PM UTC

jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

So in addition to updating the robots.txt file, which really only blocks a small number of them.

Seems CF has been gathering data and profiling these malicious agents.

This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-blo...

Basically becomes a game of cat and mouse.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#22

Post by colechristensen » Wed, Jul 02, 2025, 2:11 PM UTC

postalcoder wrote: post #11

> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets…

>doesn't punishing respectful and transparent bots only incentivize obfuscation?

They're cloudflare and it's not like it's particularly easy to hide a bot that is scraping large chunks of the Internet from them. On top of the fact that they can fingerprint any of your sneaky usage, large companies have to work with them so I can only assume there are channels of communication where cloudflare can have a little talk with you about your bad behavior. I don't know how often lawyers are involved but I would expect them to be.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#23

Post by preachermon » Wed, Jul 02, 2025, 2:12 PM UTC

alganet wrote: post #13

> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.

[flagged]

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#24

Post by jerf » Wed, Jul 02, 2025, 2:13 PM UTC

postalcoder wrote: post #11

> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets…

"doesn't punishing respectful and transparent bots only incentivize obfuscation?"

Sure, but we crossed that bridge over 20 years ago. It's not creating an arms race where there wasn't already one.

Which is my generic response to everyone bringing similar ideas up. "But the bots could just...", yeah, they've been doing it for 20+ years and people have been fighting it for just as long. Not a new problem, not a new set of solutions, no prospect of the arms race ending any time soon, none of this is new.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#25

Post by rorylaitila » Wed, Jul 02, 2025, 2:15 PM UTC

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way.

"Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding.

The big challenge is discoverability. Before, information arbitrage was one pathway to get your content discovered, or to skim a profit. This is over with AI. New means of discovery are necessary, largely network and community based. AI will throw you a few bones, but it will be 10% of what SEO did.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#26

Post by hombre_fatal » Wed, Jul 02, 2025, 2:15 PM UTC

postalcoder wrote: post #11

> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets…

Next line:

> The rule has also been expanded to include more signatures of AI bots that do not follow the rules.

The Block AI Bots rule on the Super Bot Fight Mode page does filter out most bot traffic. I was getting 10x the traffic from bots than I was from users.

It definitely doesn't rely on robots.txt or user agent. I had to write a page rule bypass just to let my own tooling work on my website after enabling it.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#27

Post by chasd00 » Wed, Jul 02, 2025, 2:15 PM UTC

lucasyvas wrote: post #18

I fail to see how this won’t just result in UA string or other obfuscation.

a crawler doesn't have to change anything, they can just ignore the robots.txt file. It's up to the client to read robots.txt and follow its directives but there's no technical reason why the client cannot just ignore everything in the file period.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#28

Post by postalcoder » Wed, Jul 02, 2025, 2:16 PM UTC

jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

This is interesting. The reasoning and response don't line up.

  > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said

  >  prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation

This seems to be targeted at taxing training of language models, but why an exclusion for the RAG stuff? That seems like it has a much greater immediate impact for online content creators, for whom the bots are obviating a click.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#29

Post by chasd00 » Wed, Jul 02, 2025, 2:19 PM UTC

cratermoon wrote: post #16

I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?

i've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#30

Post by GrayShade » Wed, Jul 02, 2025, 2:21 PM UTC

cratermoon wrote: post #16

I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?

Yes, they have over the years, for example https://blog.cloudflare.com/residential-proxy-bot-detection-..., https://blog.cloudflare.com/cloudflare-bot-management-machin..., https://blog.cloudflare.com/introducing-bot-analytics/.