Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#291

Post by selfhoster11 »

johanyc wrote: post #275
ashdksnndck wrote: post #216

Earlier quoted context omitted.

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

> the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content […] > When someone on the internet tries to sell you something for a dollar, how often do you really take them up on it? It depends on how micro they are. Your example of $1 is quite big. It should be cents or even less. Several examples. When using chatgpt api, do you…

$1 is not a micro-transaction, it's just a regular transaction. It's not micro until it's only several cents at most.

I'm not the parent, but when I use the OpenAI API (not ChatGPT API, that stuff is very cheap in comparison), I do keep an eye on my spend. For the more variable models like o3, o3-pro, GPT-4.5, expenditure can quickly exceed what you decided to spend. I'm glad you can set spending limits if you choose to do so.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#292

Post by mariconrobot »

eddythompson80 wrote: post #272
ashdksnndck wrote: post #264

Earlier quoted context omitted.

I think the costs of serving digital content are so low that you don’t need to rely on oversubscription. The user can stream 24/7 and you would still make money (assuming you got the working IP payment model like YouTube does that divide up individual user revenue proportional to that user’s watch time - I think Spotify has a problem here). The only “anti abuse” you need is to enforce that the user only streams one t…

The "cost" I was referring to is the cost to produce the content you are paying for. Not the cost to distribute it. Of course the cost to distribute it digitally is negligible (hell, lets assume it's $0), but the cost to produce it isn't even remotely close to that. > The only “anti abuse” you need is to enforce that the user only streams one thing at a time. We're talking about a flat fee you pay that gives you "acc…

[dead]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#293

Post by nosioptar »

Tijdreiziger wrote: post #276
nosioptar wrote: post #210

Earlier quoted context omitted.

A week. I'm constantly getting cloudflare nonsense that thinks I'm a bot. (Boring firefox + ublock setup.) I wouldn't be surprised if I start seeing a screen trying to get me to pay. If so, I'll do what I currently do when asked to do a recaptcha, I fuck off and take my business elsewhere.

Are you behind a CGNAT?

Not that I know of.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#294

Post by lofaszvanitt »

Oh dear god, the gatekeepers are turning evil. Just watch as they devour the internet. Cloudflare is slowly losing its mask, and soon we’ll be staring into the jaws of the deadly spider.

We protec you from evil depths of the inernet, like ddos.

We protecc you from evil AI.

Now we handle payments for you, of course you have to use our network.

...

...

...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#295

Post by lofaszvanitt »

imiric wrote: post #46
asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Because this must be done on the gov level.

As I've written about this before: https://news.ycombinator.com/item?id=40972106

Re: Cloudflare to introduce pay-per-crawl for AI bots

#296

Post by lofaszvanitt »

vasilzhigilei wrote: post #123

Man, HN is sleeping on this right now. This is huge. 20% of the web is behind Cloudflare. What if this was extended to all customers, even the millions of free ones? Would be really amazing to get paid to use Cloudflare as a blog owner, for example

Nah this is a very wrong way to let AI shills take your unique content away for morsels.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#297

Post by lofaszvanitt »

9283409232 wrote: post #115
nottorp wrote: post #94

So we used to have this company that did good things for the internet... like usable search... Now we have this company that does good things for the internet... like ddos protection, cdns, and now protecting us from "AI"... How long will the second one last before it also becomes universally hated?

Cloudflare isn't universally hated but I think most people are very nervous about the power Cloudflare holds. Bluesky puts it best "the company is tomorrow's adversary" and Cloudflare is turning into a powerful adversary.

Yeah, time to kneecap them before it's too late. EU is sleeping...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#298

Post by dhx »

rickdeckard wrote: post #141
dhx wrote: post #112

Earlier quoted context omitted.

It depends on the content producer. I would argue the best resourced content producers (governments and large companies) are incentivised to give AI bots as much curated content as possible that is favourable to their branding and objectives. Even if it's just "soft influence" such as the French government feeding AI bots an overwhelming number of articles about how the Eiffel Tower is the most spectacular tourist at…

> It depends on the content producer. I would argue the best resourced content producers (governments and large companies) are incentivised to give AI bots as much curated content as possible that is favourable to their branding and objectives. Yeah, if the content being processed is NOT the product being sold by the creator. > [..] the report makes fairly clear some of the BBC's motivation for global reach that shou…

> What kind of monetization model would this be for BBC?

BBC is taxpayer funded by Britons who have elected British officials convinced of the benefits of using taxpayer funds to influence perception (or even just prevent misinformation) of Britain and Britons, using BBC's global reach.

That is, BBC is not required to completely balance the books with consumer payments received in exchange for entertainment. British taxpayers are also part funding the BBC to ensure the soft diplomacy benefits of the BBC continue to be enjoyed by British people.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#299

Post by troyvit »

__MatrixMan__ wrote: post #233
gessha wrote: post #120

Earlier quoted context omitted.

But why exactly does it have to be on an append-only ledger where transactions are processed/validated for a fee? Why can’t it be a more conventional transaction processor like VISA on top of the banking system?

Because conventional transaction processors can be compelled to shut off payments to publishers whose content offends the powerful. Just look at what happened to wikileaks.

Seriously. Conventional transaction processors are the problem, not the solution. They can pick favorites and shut off payments for non-favorites, change their TOS, hit you with surprise transaction fees, randomly suspend your account, insert middlemen, etc. etc.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#300

Post by SparkyMcUnicorn »

kentonv wrote: post #246
SparkyMcUnicorn wrote: post #209

Earlier quoted context omitted.

How can you tell the difference, in a way that can't be spoofed? This is a genuine question, since I see you work at CF. I'm very curious what the distinction will be between a user and a crawler. Is trust involved, making spoofing a non-issue?

I don't personally work on bot detection, and I don't know exactly what techniques they use. But if you think about it: crawlers are probably not hard to identify, as they systematically download your entire web site as well as every other site on the internet (a significant fraction of which is on Cloudflare). This traffic pattern is obviously going to look extremely different from a web browser operated by a human.…

Cross-site traffic was completely overlook in my previous comment. Good point, and definitely changes the perspective a bit.
Post reply on HN