Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

41–50 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#41

Post by alganet »

preachermon wrote: post #23
alganet wrote: post #13

> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.

[flagged]

These kinds of comparisons rarely lead to good discussions.

Let's instead be focused and talk about real stuff.

Consider https://learnpythonthehardway.org/ for example. It has influenced a generation of Python developers. Not just the main website, but the tons of Python code and Python-related content it inspired.

Why would anyone write these kinds of textbooks/websites/guides if AI can replace them? AI companies are effectively broadcasting you don't need the hard way anymore, you can just vibe.

Arguibly though, without the existance of Learn Python the Hard Way and similar content, AI would be worse at writing Python stuff. That's what I mean by "main source of food", good content that influences a lot of people. Net-positive effects hard to predict or even identify except for the more popular cases (such as LPTHW).

If my prediction is right, no one will notice that good content has stopped being produced. It will appear as if content is being created in generally the same way as before, but in reality, these long tail initiatives like LPTHW will have ceased before anyone can do anything about it.

Again, I don't see a way out of this scenario. Not for AI companies, not for content writers. This is going to happen. The world in which I'm wrong is the best one.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#43

Post by mschuster91 »

Sol- wrote: post #9

Do the major AI companies actually honor robots.txt? Even if some of their publicly known crawlers might do it, surely they have surreptitious campaigns where they do some hidden crawling, just like how they illegally pirate books, images and user data to train on.

Cloudflare, for all I hate their role as a gatekeeper these days, actually has the leverage to force the AI companies to bend.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#44

Post by postalcoder »

fennecfoxy wrote: post #31
postalcoder wrote: post #28

Earlier quoted context omitted.

This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This…

With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Goo…

I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#45

Post by rorylaitila »

fennecfoxy wrote: post #33
rorylaitila wrote: post #25

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information a…

>AI contains all published human information No, it most certainly does not. It was certainly trained on large swathes of human knowledge/interactions. A model that consists of a perfect representation/compression of all this info is a zip file, not a model file.

AI providers have scrapped and will continue to, all internet published information or virtually so. Since "Information" is infinite, AI cannot contain "all information" in a complete sense. But it certainly answers almost everything that matters for any existing search query that has ever been targeted by a webpage that is crawlable.

In any case, as manifest by real world SEO, which is plummeting in traffic for informational queries, the effect is the same. This real world impact is what matters and will not be reversed, regardless of attempts at blocking.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#46

Post by spwa4 »

alganet wrote: post #13

> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.

Yes what everyone wants to do with AI: generate entertainment and interactions with humans, including economical ones, will need to happen or AI will starve.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#47

Post by dawnerd »

I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically.

It also completely put a stop to perplexity as far as I can tell.

And the robots file meant nothing, they’d still request it hundreds of thousands of times instead of caching it. Every request they’d hit it first then hit their intended url.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#49

Post by rorylaitila »

ozgrakkurt wrote: post #38
rorylaitila wrote: post #25

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information a…

You are assuming LLMs will replace search engines. Why is this the case? To me it seems like there has to be so much optimization for this to happen that, it is not likely. LLM answers are slow and unreliable. Even using something like perplexity doesn’t give much value over using a regular search engine in my experience

LLMs will not fully replace search engines, but Google and Bing are evolving to be LLM first, anyhow. So "what is a search engine" today is not what it was yesterday. Let's call the time before LLMs, traditional search. LLM first products bundle some aspect of traditional search. And traditional search is adding LLM answers.

Traditional search will still be highly useful for transactional, product, realtime, and action oriented queries. Also for discovering educational/entertainment content that is valued in of itself and cannot be reformulated by LLM.

Post reply on HN