Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.
Cloudflare to introduce pay-per-crawl for AI bots
11–20 of 317 posts
Re: Cloudflare to introduce pay-per-crawl for AI bots
#12And all of this does not stop the incumbents who have already stolen everything.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#13All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#14Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.
CF is acting as the merchant of record, so they will be the ones billing, it's unclear what cut of the price they will take (if any) or if they will include it in their bundled services.
This should be expanded to allow for:
* micropayments and subscriptions
* integration with the browser UI/UX
* multiple currencies
* implementation of multiple payment systems, including national instant settlement systems like UPI, NPP, FedNow etc.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#15I would totally do this. "Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis. Enshittify the enshittification machine. We should also throw ads in there, via a deliberate prompt injection that the AI companies expose through an API. I totally won't misuse it ;)
Do you think that there is $25 of value in the creation of your blog, to say nothing of value that AI may be able to extract from it? (I'm speaking hypothetically, as I haven't looked at your profile to see if you link your blog, but I will do so now.)
Edit: I have checked, and I've read your blog before. I think the answer to the question depends on who is asking but I don't know how you feel about the matter. I think asking for folks to pay for free things is a different value proposition than a pay-per-use fee, so the economics are different. You're also offering something different when you give away a blog and monetize access to a community or something similar, which is different still to accepting donations and so on. I don't know what you do for work or if you do your blog full time, but I think it's cool that you make it all the same.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#16All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#17I would totally do this. "Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis. Enshittify the enshittification machine. We should also throw ads in there, via a deliberate prompt injection that the AI companies expose through an API. I totally won't misuse it ;)
I mean, I wouldn't care who referred the paying user. I would optimize my blog to serve free and paying users to the best of my ability. I would hope that I could do that but I don't know what you are trying to do with your blog. Most blogs could probably be self-hosted with some caching and static layout where possible to perhaps avoid needing to use Cloudflare. I guess you already have to be using CF to have access…
Some of my blog posts are linked by programming language docs for cryptography. Others have helped queer folks transition to a higher-paying tech career.
It's difficult to quantify either of those things in a dollar value. I've opted to not ever do so. But if I can make the AI slop machines pay to view my furry/tech ramblings, I will do enthusiastically.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#18Soon they could decide if your requests come from a specific company IP or networks, because you look suspicious...
In addition, bot fighting was never supposed to be about blocking automatic users but about blocking abusers, like spammers and co. So now it means that bad actor can have a free pass if they pay (with stolen credit cards...)
What I think would have been more fair is to propose rate limiting that would apply the same to everyone, so website should be reasonable in the limit they set for the normal users to not be annoyed. And then, you could pay to be able to have higher rate limit to ressources. That will compensate for the incurred cost to the infrastructure and the website owner. With that cloudflare could be in a good position to controle the rate limit, negotiate and collect payments to give it to the website owners.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#19All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.
Yes, but I do feel this makes "theft" arguments stronger if they're deliberately evading the paywall, if you decided to be litigious about it.