Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

41–50 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#41

Earlier quoted context omitted.

Cloudflare has not used CAPTCHAs since 2023: https://blog.cloudflare.com/turnstile-ga/

To save a visit: they use turnstile, a captcha replacement. The checkbox with verify you are a human. I would call that a captcha, but it is debatable if a non-puzzle check is.

If I say CloudFlare captcha and you know what I mean, does it really matter?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#42
So:

1. Encourage fencing off everything by default to maximize need for bypass

2. Offer bypass through payment

3. Profit!

You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understand the problem, they will fix it after a few months just before the next cheapest contractor is hired and we start all over again.

Not saying Cloudflare is just an extortion racket, but it's getting closer by the day.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#43
post #5

Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.

Is this companies collecting data for model training, or is it agentic tools operating on behalf of users?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#45

"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee. And all of this does not stop the incumbents who have already stolen everything.

[deleted]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#46
post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and regulate these companies just like we did with Big Tobacco.

Brave's BAT is also a good attempt at fixing this, but x402 seems like a more generic solution. It's a shame that neither has any chance of gaining traction, partly because of the cryptocurrency stigma, and partly because of adtech's tight grip on the current web.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#47

So: 1. Encourage fencing off everything by default to maximize need for bypass 2. Offer bypass through payment 3. Profit! You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understa…

[deleted]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#48

I would totally do this. "Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis. Enshittify the enshittification machine. We should also throw ads in there, via a deliberate prompt injection that the AI companies expose through an API. I totally won't misuse it ;)

I mean, I wouldn't care who referred the paying user. I would optimize my blog to serve free and paying users to the best of my ability. I would hope that I could do that but I don't know what you are trying to do with your blog. Most blogs could probably be self-hosted with some caching and static layout where possible to perhaps avoid needing to use Cloudflare. I guess you already have to be using CF to have access…

> Do you think that there is $25 of value in the creation of your blog, to say nothing of value that AI may be able to extract from it?

I think the more pertinent question is: Can an AI company determine the value of that content automatically without seeing it? Because if they can’t, why would they pay for it?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#49

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

And we are back to: how much would you pay for a website before you access said website for the first time?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#50
post #49

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

And we are back to: how much would you pay for a website before you access said website for the first time?

The Web Monetisation protocol solved this by doing a prorata based on how long you spent on the page.
Post reply on HN