Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#51

Post by soatok »

JimDabell wrote: post #48
aspenmayer wrote: post #15

Earlier quoted context omitted.

I mean, I wouldn't care who referred the paying user. I would optimize my blog to serve free and paying users to the best of my ability. I would hope that I could do that but I don't know what you are trying to do with your blog. Most blogs could probably be self-hosted with some caching and static layout where possible to perhaps avoid needing to use Cloudflare. I guess you already have to be using CF to have access…

> Do you think that there is $25 of value in the creation of your blog, to say nothing of value that AI may be able to extract from it? I think the more pertinent question is: Can an AI company determine the value of that content automatically without seeing it? Because if they can’t, why would they pay for it?

If they won't pay for it, they can also kindly fuck off and not crawl my blog. Both are fine outcomes for me.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#52

Post by aspenmayer »

JimDabell wrote: post #48
aspenmayer wrote: post #15

Earlier quoted context omitted.

I mean, I wouldn't care who referred the paying user. I would optimize my blog to serve free and paying users to the best of my ability. I would hope that I could do that but I don't know what you are trying to do with your blog. Most blogs could probably be self-hosted with some caching and static layout where possible to perhaps avoid needing to use Cloudflare. I guess you already have to be using CF to have access…

> Do you think that there is $25 of value in the creation of your blog, to say nothing of value that AI may be able to extract from it? I think the more pertinent question is: Can an AI company determine the value of that content automatically without seeing it? Because if they can’t, why would they pay for it?

If they have lots of reasons to believe it could be relevant due to requests for content by name or reference, as well as citations and other knowledge graph links, I think they could run some kind of A/B test to see what the market will bear based on what they estimate the crawls per billing period would be.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#53

Post by sofixa »

delusional wrote: post #19
some_furry wrote: post #16

Earlier quoted context omitted.

Yes, but I do feel this makes "theft" arguments stronger if they're deliberately evading the paywall, if you decided to be litigious about it.

This isn't the kind of problem that really ought to be solved through courts. It's obvious to anyone that this is a new kind of problem that no author of the current jurisprudence envisioned. We need new legislation to stop this kind of abuse of the commons.

Yes. It's always weird to me that people expect laws written over centuries, using precedents from even more centuries, to be able cover scenarios their authors couldn't have possibly imagined.

Civil law countries seem better at keeping their laws up to date with new threats whereas a few common law ones (most notably the US) really insist on digging through what an 18th century slave owner would have thought about e.g. AI.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#54

Post by mariusor »

OtherShrezzing wrote: post #44

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

Everyone seems to hate on Brave for trying something like this. Granted they use shady cryptocurrency instead of a one time fee, or subscription model, but what the hell...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#55

Post by mdrzn »

sofixa wrote: post #50
mdrzn wrote: post #49

Earlier quoted context omitted.

And we are back to: how much would you pay for a website before you access said website for the first time?

The Web Monetisation protocol solved this by doing a prorata based on how long you spent on the page.

What if I save the page locally on my machine? Or archive it?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#57

Post by imiric »

OtherShrezzing wrote: post #44

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

Brave already does this with BAT[1].

It's a shame that it will never gain traction. Partly because of the cryptocurrency stigma, some missteps by Brave Inc. that have tarnished its credibility, and partly because of adtech's tight grip on the web.

[1]: https://basicattentiontoken.org/

Re: Cloudflare to introduce pay-per-crawl for AI bots

#58

Post by quixoticaxolotl »

crgwbr wrote: post #10

All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.

The blog post covers this. The announcement also drops relying on spoofable user agents for crawler identification and requires crawlers to voluntarily identify themselves via RFC 9421 cryptographic message signatures to get access: https://blog.cloudflare.com/introducing-pay-per-crawl/#payme...

There are likely incentives for AI companies to try to simulate human users as much as possible, but the value proposition here is that CF is so good at identifying and stopping those that signing a request becomes the path of least resistance.

Disclosure: I am on the team that wrote the RFC 9421 message signature implementation at Cloudflare and its use in the pay per crawl project. A separate blog post went out here: https://blog.cloudflare.com/verified-bots-with-cryptography/

Re: Cloudflare to introduce pay-per-crawl for AI bots

#59

Post by graeme »

JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

If the traffic pays anything at all it's trivial to fund the infrastructure to handle the traffic. Historically sites have scaled well under traffic load.

What's happened recently is either:

1. More and more sites simply block bot, scrapers etc. Cloudflare is quite good at this or

2. Sites which can't do this for access reasons or don't have a monetization model and so can't pay to do it get barraged

IF this actually pays, then it solves a lot of the problems above. It may not pay publishers what they would have earned pre-ai, but it should go a long way to addressing at the very least the costs of a bot barrage and then some on top of that.

Post reply on HN