Cloudflare's new AI traffic options for customers

Post by tempest_ » Sun, Jul 26, 2026, 2:44 PM UTC

AnthonyMouse wrote: post #102

dannyw wrote: post #43

Earlier quoted context omitted.

Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly. That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a tr…

The irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?

Post by schainks » Sun, Jul 26, 2026, 3:18 PM UTC

siva7 wrote: post #104

inigyou wrote: post #90

Earlier quoted context omitted.

Their scraping products are mostly about paying them money to not be blocked.

So Google has to pay Cloudflare 10B$ to get Googlebot moved to their default Allowlist otherwise Google would loose access to a significant portion of the internet? Genius move.

Post by matheusmoreira » Sun, Jul 26, 2026, 4:53 PM UTC

akersten wrote: post #68

matheusmoreira wrote: post #36

Earlier quoted context omitted.

> Please consider installing one of the many PoW schemes such as anubis Why not go all the way and mine monero instead of just completely wasting the work?

because then someone will complain that you're stealing their CPU cycles or something, on the website they chose to visit and execute

Post by expedition32 » Sun, Jul 26, 2026, 7:01 PM UTC

altairprime wrote: post #123

timpera wrote: post #109

Earlier quoted context omitted.

Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.

As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers. Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones. Certainly, I…

Post by AnthonyMouse » Sun, Jul 26, 2026, 7:47 PM UTC

tempest_ wrote: post #141

AnthonyMouse wrote: post #102

Earlier quoted context omitted.

The irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?

Search traffic is nose diving due to LLM use. The fundamental calculus with google is you install GA and it helps your SEO has changed and google is riding out what will eventually wither as people start to reevaluate the trade off.