Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

271–280 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#271
Very neat idea to charge the crawlers via headers.

Wondering if it blocks things such as Infatica as well, where AI scrapers use that service to scrape data behind residential proxies, in which it's impossible to get the actual residential proxy for.

App developers hide Infatica drone code in things; I haven't thought of a reasonable way to block it.

IP addresses use the systems, humans use them in certain ways, Cloudflare must have a decent way to block these.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#272

Earlier quoted context omitted.

It's not an ideal solution because any fixed cost solution is begging for a middle man/reseller to be introduced. Like I pay the $5 monthly flat fee (or $500, $5k, $500k, whatever it's known fixed cost for me) for the system, turn around and resell all content for a $1 monthly flat fee. There is a real cost to the content you're consuming with that flat-fee. So either the flat fee is more of "credit" system or it's r…

I think the costs of serving digital content are so low that you don’t need to rely on oversubscription. The user can stream 24/7 and you would still make money (assuming you got the working IP payment model like YouTube does that divide up individual user revenue proportional to that user’s watch time - I think Spotify has a problem here). The only “anti abuse” you need is to enforce that the user only streams one t…

The "cost" I was referring to is the cost to produce the content you are paying for. Not the cost to distribute it. Of course the cost to distribute it digitally is negligible (hell, lets assume it's $0), but the cost to produce it isn't even remotely close to that.

> The only “anti abuse” you need is to enforce that the user only streams one thing at a time.

We're talking about a flat fee you pay that gives you "access to content on the internet".

Oh yeah? How does one "stream" an article? Does playing a video at 2x make it 1/2 price? what about 1000x?

Ok, ok, Lets steelman this argument and assume we come up with resonable common sense answers to all these questions. "an article counts as x minutes". "limit playback to max of 2x and figure out some reasnonable formula to pay the creator", etc

Congratulations, you've invited a credit system with extra steps. The "flat fee" is actually the fee for (602430) * 2 minutes a month. One could "Donate Minutes" left on their account at the end of the month to their favorite creator. well, instead of trading them to your favorite creator, why don't you trade them in for $$

> Given the marginal cost of distributing the good stuff is the same as the bad stuff, why make the customer feel bad about watching by adding an incremental cost? Just let it rip. If you have a lot of good stuff, customers will be willing to pay more for the bundle. Once they’re in the bundle, let them watch exactly what they want.

The model works for YouTube because of the centralized nature of YouTube. I think that model can work for other centralized systems too like cloudflare.

Hell, maybe that was cloudflare endgame all along. If a good chunk of the internet is running behind cloudflare proxies, then cloudflare could do brave's BAT idea but actually sanely with like a normal payment and subscription etc.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#273

They're going to need to deal with camoufox being easily able to circumvent their bot detection before they start charging for this, imho.

Haven't heard of / seen camoufox; makes sense to detect browsers by running a modified firefox that renders things like WebGL/Canvas and makes it incredibly hard to stop them.

This coupled with advanced proxy lists, which is what AI companies, or rather just random scrapers will go for if they're blocked and charged.

They want the data for free.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#274

Earlier quoted context omitted.

If I understand correctly, it will be not by creating a new iteration, but by destroying the current one.

We are in agreement that AI will destroy the current one. I don't see how the new iteration that AI would produce would have "more genuine information" seeing as how LLMs are just predicting what word follows the previous word. How is that genuine?

The current state of the internet brought the Fediverse, a new breed of social network(s) that, by design, has evaded corporate/government/salesmen capture so far.

Nevertheless, I can't say if the Fediverse will become irreversibly captured using new tactics. If that happens, a new iteration will happen.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#275
post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

> the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content […] > When someone on the internet tries to sell you something for a dollar, how often do you really take them up on it?

It depends on how micro they are. Your example of $1 is quite big. It should be cents or even less.

Several examples. When using chatgpt api, do you really worry how much a short q&a session will cost you? Do you stress whether to turn on the light in your room or not (electricity cost is also micro-transaction if you think about it)?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#276

How long before we get pay per browse and the internet is 6ft under?

A week. I'm constantly getting cloudflare nonsense that thinks I'm a bot. (Boring firefox + ublock setup.) I wouldn't be surprised if I start seeing a screen trying to get me to pay. If so, I'll do what I currently do when asked to do a recaptcha, I fuck off and take my business elsewhere.

Are you behind a CGNAT?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#277

Earlier quoted context omitted.

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

> I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. > The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. You just described bundling - that’s how…

I want to have single subscription and then happy to pay extra $5 to watch an out of network show.

I feel streaming companies should offer some sort of content exchange for a fee so users wouldn’t have to switch platforms.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#278
post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

Except I don't want to use crypto, I don't want to accept crypto for content, I don't want to pay middlemen for using crypto.

Micro payments using crypto is just a way for folks to prop up crypto currencies. It also is a dead concept, because how do we all agree on _which_ crypto to use? If I'm browsing the internet, and each site only accepts a particular shit coin, is that ok? Does everyone just use a single stablecoin? Now everything is locked to a single currency?

The cloudflare approach is honestly ideal, because it charges people profiting from your content, not humans looking to read your content. It also doesn't use crypto.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#279

There’s a tiny but important difference between scraping and getting work done. Scraping is associate with mindless extraction. Like a vacuum cleaner sucking-in any data without context, permission, or contributing value back. On the other hand AI agents aren’t here to scrape for the sake of it - I have seen it first hand. They are here to get work done, mostly researching, summarizing, assisting, building new produc…

Funny finding you here. ;) Hey Orlie.

Hey datarade

Re: Cloudflare to introduce pay-per-crawl for AI bots

#280

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Not sure how Google is winning AI, at least from the sophisticated consumer's perspective. Their AI overviews are often comically wrong. Sure, they may have Good APIs for their AI, and good technical quality for their AIs, but for the general user, their most common AI presentation is woefully bad.

I assume the high volume of search traffic forces Google to use a low quality model for AI overviews. Frontier Google models (e.g. Gemini 2.5 pro) are on-par, if not 'better', than leading models from other companies.
Post reply on HN