Cloudflare to introduce pay-per-crawl for AI bots • Hacker News Board

Re: Cloudflare to introduce pay-per-crawl for AI bots

#121

Post by squigz » Tue, Jul 01, 2025, 1:38 PM UTC

imiric wrote: post #46

asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#122

Post by 9283409232 » Tue, Jul 01, 2025, 1:38 PM UTC

PeterStuer wrote: post #42

So: 1. Encourage fencing off everything by default to maximize need for bypass 2. Offer bypass through payment 3. Profit! You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understa…

I don't trust Cloudflare but this is not a problem they created. They solved a real problem with DDoS protection in the beginning and now AI crawlers increasing server cost is not a negligible problem. The CEO of iFixit called out Anthropic publicly for hitting their site a million times in 24 hours to scrape it. We are passed the point of good faith action from these AI companies. They are adversarial and need to be treated as such.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#123

Post by vasilzhigilei » Tue, Jul 01, 2025, 1:39 PM UTC

Man, HN is sleeping on this right now. This is huge. 20% of the web is behind Cloudflare. What if this was extended to all customers, even the millions of free ones? Would be really amazing to get paid to use Cloudflare as a blog owner, for example

Re: Cloudflare to introduce pay-per-crawl for AI bots

#124

Post by mattlondon » Tue, Jul 01, 2025, 1:40 PM UTC

kevlened wrote: post #65

OtherShrezzing wrote: post #44

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

Google had an experiment called Google Contributor where you could buy all your own ads. This effectively had the experience you're describing (prepay and get no ads until it runs out). They tried it twice, so someone wanted it to work. I was always curious why they shuttered it.

Probably because people didn't want to pay. It's easy and cheap to say "I'd pay X to access website without ads!" ...but when it came to it and people had that option, essentially no-one did it.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#125

Post by mysteria » Tue, Jul 01, 2025, 1:41 PM UTC

mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Even before AI was a thing some websites would deny all crawlers in robots.txt except for the Googlebot for the same reason.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#126

Post by kentonv » Tue, Jul 01, 2025, 1:42 PM UTC

FloatArtifact wrote: post #64

What about if somebody uses artificial intelligence crawler to help them navigate the web as an accessibility tool? Enabling UI automation. It already throws up a lot of... uh... troublesome verifications.

How would an individual user use a "crawler" to navigate the web exactly? A browser that uses AI is not automatically a "crawler"... a "crawler" is something that mass harvests entire web sites to store for later processing...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#127

Post by gessha » Tue, Jul 01, 2025, 1:43 PM UTC

danaris wrote: post #82

asimpletune wrote: post #81

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot. I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the mod…

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in. "Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to t…

The commons are not destined to become a tragedy and they can become a long-term resource everyone can enjoy[1]. You need clear boundaries, reliable monitoring of shared resource, reasonable balance between costs and benefits, etc.

> I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to this company

Or weasel out of being a non-profit.

[1] https://aeon.co/essays/the-tragedy-of-the-commons-is-a-false...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#128

Post by dboreham » Tue, Jul 01, 2025, 1:44 PM UTC

gessha wrote: post #120

trollbridge wrote: post #67

Earlier quoted context omitted.

Something like BAT isn't that wasteful, and without crypto you'd be stuck never getting paid by bad actors in the scheme.

But why exactly does it have to be on an append-only ledger where transactions are processed/validated for a fee? Why can’t it be a more conventional transaction processor like VISA on top of the banking system?

Because Visa doesn't (hasn't) wanted to do microtransactions, ever, since the beginning of the internet.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#129

Post by krab » Tue, Jul 01, 2025, 1:44 PM UTC

cedws wrote: post #78

Can someone explain the payment headers part? Why not just have a header called X-Crawl-Key or something and intercept that header to figure out who to charge for the request?

They have some headers for authentication. The payment part is for the price negotiation. The headers tell you that Cloudflare wants to charge you for this particular content and you tell CF that you're OK with being charhed up to $AMOUNT.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#130

Post by dboreham » Tue, Jul 01, 2025, 1:47 PM UTC

asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

As someone who has actually built working micro payments systems, this was of interest. Worth noting though that it's really just "document-ware" -- there's no code there[1], and their proposed protocol doesn't look like it was thought through to the point where it has all the pieces that would be needed.

[1] E.g. this file is empty: https://github.com/coinbase/x402/blob/main/package.json