Cloudflare to introduce pay-per-crawl for AI bots

#1

Post by scotchmi_st » Tue, Jul 01, 2025, 10:20 AM UTC

Cloudflare to introduce pay-per-crawl for AI bots
blog.cloudflare.com

Re: Cloudflare to introduce pay-per-crawl for AI bots

#2

Post by yodon » Tue, Jul 01, 2025, 10:45 AM UTC

Currently in private beta

Re: Cloudflare to introduce pay-per-crawl for AI bots

#3

Post by Toritori12 » Tue, Jul 01, 2025, 10:45 AM UTC

Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#4

Post by yantramanav » Tue, Jul 01, 2025, 10:46 AM UTC

While this is a neat idea, how does it negate all the data theft being done by the bots so far?

I recently saw a research paper behind a paywall but ChatGPT readily gave me a detailed summary of that article. I’m afraid the cat's out of the bag now.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#5

Post by suyash » Tue, Jul 01, 2025, 10:46 AM UTC

Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#6

Post by some_furry » Tue, Jul 01, 2025, 10:50 AM UTC

I would totally do this.

"Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis.

Enshittify the enshittification machine.

We should also throw ads in there, via a deliberate prompt injection that the AI companies expose through an API. I totally won't misuse it ;)

Re: Cloudflare to introduce pay-per-crawl for AI bots

#7

Post by teruakohatu » Tue, Jul 01, 2025, 10:51 AM UTC

yantramanav wrote: post #4

While this is a neat idea, how does it negate all the data theft being done by the bots so far? I recently saw a research paper behind a paywall but ChatGPT readily gave me a detailed summary of that article. I’m afraid the cat's out of the bag now.

All the LLMs are being trained on LibGen/Anna’s Archive so it’s not in the least surprising they can tell you about papers behind paywalls.

But I don’t think they are creating accounts to scrape paywalled data from the original source itself.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#8

Post by jgrahamc » Tue, Jul 01, 2025, 10:51 AM UTC

yodon wrote: post #2

Currently in private beta

https://techcrunch.com/2025/07/01/cloudflare-launches-a-mark...

"Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default in support of the company’s broader goal of a “permission-based approach to crawling.”"

Re: Cloudflare to introduce pay-per-crawl for AI bots

#9

Post by aspenmayer » Tue, Jul 01, 2025, 10:52 AM UTC

Related from TC:

> Cloudflare launches a marketplace that lets websites charge AI bots for scraping

https://techcrunch.com/2025/07/01/cloudflare-launches-a-mark...

https://archive.is/6UDUv

Re: Cloudflare to introduce pay-per-crawl for AI bots

#10

Post by crgwbr » Tue, Jul 01, 2025, 10:53 AM UTC

All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.