Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

31–40 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#31

Post by fennecfoxy »

postalcoder wrote: post #28
jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This…

With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money.

It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page.

A blog author could not complain that their blog is getting ragged when they're extremely liable to be Google/whatever searching all day and basically consuming others' content in exactly the same way that they're trying to disparage.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#32

Post by Meekro »

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference. Why is everyone so upset?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#33

Post by fennecfoxy »

rorylaitila wrote: post #25

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information a…

>AI contains all published human information

No, it most certainly does not. It was certainly trained on large swathes of human knowledge/interactions.

A model that consists of a perfect representation/compression of all this info is a zip file, not a model file.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#34

Post by kube-system »

lucasyvas wrote: post #18

I fail to see how this won’t just result in UA string or other obfuscation.

Cloudflare’s filtering is already way more sophisticated than just looking at UA string or other voluntary reporting. They’re almost certainly using fingerprinting and behavioral analytics.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#35

Post by fennecfoxy »

alganet wrote: post #13

> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.

Additionally, ad blocker usage is apparently at 30%. So it's a redundant or more nuanced argument, really.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#36

Post by JimDabell »

blakesterz wrote: post #4

The list of bots is pretty short right now: https://developers.cloudflare.com/bots/concepts/bot/#ai-bots

> AI bots

> You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM).

> CCBot (Common Crawl)

Common Crawl is not an AI bot:

https://commoncrawl.org

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#37

Post by GrayShade »

btown wrote: post #10

The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.

> sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare

Do you have a source for that? https://blog.cloudflare.com/content-independence-day-no-ai-c... does say "changing the default".

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#38

Post by ozgrakkurt »

rorylaitila wrote: post #25

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information a…

You are assuming LLMs will replace search engines. Why is this the case?

To me it seems like there has to be so much optimization for this to happen that, it is not likely. LLM answers are slow and unreliable. Even using something like perplexity doesn’t give much value over using a regular search engine in my experience

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#39

Post by deepsiml »

Meekro wrote: post #32

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…

Not much into that kind of DevOps. What is a good basic caching in this instance?
Post reply on HN