Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

81–90 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#81

Post by mattcollins »

GrayShade wrote: post #37
btown wrote: post #10

The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.

> sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare Do you have a source for that? https://blog.cloudflare.com/content-independence-day-no-ai-c... does say "changing the default".

"This feature is available to all customers, meaning anyone can enable this today from the Cloudflare dashboard."

https://blog.cloudflare.com/control-content-use-for-ai-train...

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#82

Post by fluidcruft »

TechDebtDevin wrote: post #53
dawnerd wrote: post #47

I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically. It also completely put a stop to perplexity as far as I can tell. And the rob…

This does nothing dude. Literally nothing. OpenAI or whoever are just going to hire people like me who dont get caught. Stop ruining the experience of users and allowing cf to fill the internet with more bloated javascript challenge pages and privacy invading fingerprinting. Stop making cf the police of the internet. We're literally handing the internet to this company on a silver platter to do MITM attacks on our pr…

They literally said it significantly reduced their server resource usage. Are you suggesting they are lying?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#83

Post by bitpush »

TechDebtDevin wrote: post #67
bitpush wrote: post #15

Earlier quoted context omitted.

It is now an adversarial relationship between aibots and website, and cloudflare is merely reacting to it. Would you say the same for ddos protection? Isn't that the same as well?

They arent doing anything. They are attempting to insert themselves into the middle of a marketplace (that doesnt exist and never will) where scrapers pay for IP. They think theyre going to profit off the bots, not protect your site. Dont fall for their scam.

What do you mean they are trying to insert themselves? If I have a website that I host with cloudflare, I (as the rightful website owner) has inserted Cloudflare in between.

It isnt CF going around saying, that's a nice website you have there. I'm gonna put myself in between.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#84

Post by mrweasel »

haiku2077 wrote: post #72
deepsiml wrote: post #39

Earlier quoted context omitted.

Not much into that kind of DevOps. What is a good basic caching in this instance?

It comes down to: 1. Use the Cache-Control header to express how to cache your site correctly ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac... ) 2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers

Just note that many AI scrapers will go to great length to do cache busting. For some reason many of them feel like they need to get the absolute latest version and don't trust your cache.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#85

Post by haiku2077 »

mrweasel wrote: post #84
haiku2077 wrote: post #72

Earlier quoted context omitted.

It comes down to: 1. Use the Cache-Control header to express how to cache your site correctly ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac... ) 2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers

Just note that many AI scrapers will go to great length to do cache busting. For some reason many of them feel like they need to get the absolute latest version and don't trust your cache.

You can use Cache Control headers to express that your own CDN should aggressively refresh a resource but always serve it to external clients from cache. It's covered in the link under "Managed Caches"

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#86

Post by roguecoder »

haiku2077 wrote: post #65
conductr wrote: post #60

Earlier quoted context omitted.

The presumption I’m already using cloudfare is a start. Is this a requirement for maintaining a simple website now?

Either that or Anubis ( https://anubis.techaro.lol/docs ), yes.

So these companies broke the internet

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#87

Post by toomuchtodo »

Boldened15 wrote: post #68
postalcoder wrote: post #44

Earlier quoted context omitted.

I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.

IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.

It's really about who is getting the value from the work of the content. If content creators of all sorts have their work consumed by LLMs, and LLM orgs charge for it can capture all the value, why should people create to have their work vacuumed up for the robot's benefit? For exposure? You can't eat or pay rent with exposure. Humans must get paid, and LLMs (foundational models and output using RAG) cannot improve without a stream of works and data humans create.

Whether you call it training or something else is irrelevant, it's really exploitation of human work and effort for AI shareholder returns and tech worker comp (if those who create aren't compensated). And the technocracy has not been, based on the evidence, great stewards of the power they obtain through this. Pay the humans for their work.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#88

Post by mrweasel »

bee_rider wrote: post #62
jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

I wonder… Google scrapes for indexing and for AI, right? I wonder if they will eventually say: ok, you can have me or not, if you don’t want to help train my AI you won’t get my searches either. That’s a tough deal but it is sort of self-consistent.

Very few people seems to be complaining that Google crashes their sites. Google also publish their crawlers IP ranges, but you really don't need to rate-limit Google, they know how to back off and not overload sites.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#89

Post by mfost »

alganet wrote: post #41
preachermon wrote: post #23

Earlier quoted context omitted.

[flagged]

These kinds of comparisons rarely lead to good discussions. Let's instead be focused and talk about real stuff. Consider https://learnpythonthehardway.org/ for example. It has influenced a generation of Python developers. Not just the main website, but the tons of Python code and Python-related content it inspired. Why would anyone write these kinds of textbooks/websites/guides if AI can replace them? AI companies ar…

In a similar vein, I remember people advocating for replacing new untrained hires with AI. After all, a competent senior engineer is needed to validate the contributions of the new hires anyway and they can do the same checking the AI code.

But then, how would you even train and replace those competent seniior engineers that do the filtering when they retire? The whole system was predicated on having a chain of new hires that gain experience in the process.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#90

Post by noodle »

Meekro wrote: post #32

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…

As someone who had some outages due to AI traffic and is now using CloudFlare's tools:

Most of my site is cached in multiple different layers. But some things that I surface to unauthenticated public can't be cached while still being functional. Hammering those endpoints has taken my app down.

Additionally, even though there are multiple layers, things that are expensive to generate can still slip through the cracks. My site has millions of public-facing pages, and a batch of misses that happen at the same time on heavier pages to regenerate can back up requests, which leads to errors, and errors don't result in caches successfully being filled. So the AI traffic keeps hitting those endpoints, they keep not getting cached and keep throwing errors. And it spirals from there.

Post reply on HN