Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

231–240 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#231

Post by __MatrixMan__ »

ashdksnndck wrote: post #216
imiric wrote: post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated.

The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it.

Then we build a culture around preferring to share content which is configured to cite its sources, and we discourage sharing anything which has an obvious source with which it doesn't share its inbound microtransactions.

We already need to do our due dilligence re: determining if an information source is trustworthy (and if its sources are trustworthy, and so on). Might as well make money flow along the same structures.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#232

Post by nitwit005 »

adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

They aren't assuming it'd never change. They're upset at it getting worse. Things getting worse is generally what makes people unhappy.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#233

Post by __MatrixMan__ »

gessha wrote: post #120
trollbridge wrote: post #67

Earlier quoted context omitted.

Something like BAT isn't that wasteful, and without crypto you'd be stuck never getting paid by bad actors in the scheme.

But why exactly does it have to be on an append-only ledger where transactions are processed/validated for a fee? Why can’t it be a more conventional transaction processor like VISA on top of the banking system?

Because conventional transaction processors can be compelled to shut off payments to publishers whose content offends the powerful. Just look at what happened to wikileaks.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#234

Post by gpm »

squigz wrote: post #150
dhx wrote: post #137

Earlier quoted context omitted.

> But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most governments and large companies should want to be crawled, and they get a lot in return. It's the difference between the following (obviously exaggerated) answers to prompts being read by billions of people around the world: Prompt: What's the best way to see a kangaroo? Response…

I'd be unsatisfied with both of those answers. 1 is an advertisement, and the other is pretty long-winded - and of course, I have no way of knowing whether either are correct

The person you replied to is about the third parties companies goal though, not the users.

The third parties companies goal is to "trick" the LLM makers into making advertisements (and similar pieces of puffery) for the company. The LLM makers goal is to... make money somehow... maybe by satisfying the users desire. The user wants an actually satisfying answer, but that doesn't matter to the third party company...

Re: Cloudflare to introduce pay-per-crawl for AI bots

#235

Post by gpm »

miohtama wrote: post #140
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Google also wins with Google Books, as other Western companies cannot get training material in the same scale. Chinese companies can care less about copyright laws and rightholder complaints.

Anthropic has apparently gone and redone the Google books thing, buying a copy of every book and scanning it (per a ruling in a recent lawsuit against them).

Re: Cloudflare to introduce pay-per-crawl for AI bots

#236

Post by reverendsteveii »

nitwit005 wrote: post #232
adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

They aren't assuming it'd never change. They're upset at it getting worse. Things getting worse is generally what makes people unhappy.

this. it's changed several times over its lifetime and every change until recently has made it a better thing for the average person to use. We're out of the discovery phase and into the encirclement and exploitation phase.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#237

Post by BlueTemplar »

__MatrixMan__ wrote: post #231
ashdksnndck wrote: post #216

Earlier quoted context omitted.

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

So, Flattr 2.0 ?

https://www.ctrl.blog/entry/flattr2.html

Re: Cloudflare to introduce pay-per-crawl for AI bots

#238

Post by __MatrixMan__ »

squigz wrote: post #121
imiric wrote: post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?

Really? They're quite connected.

If the architecture of the web changes to one where people only see content that they've asked to see, and that kills advertising, it would also put a significant damper on anyone else whose business involves injecting unwanted content into a viewer's consciousness. Propagandists are the first to come to mind.

If it can become prohibitively expensive to sway an election by tampering with people's information, then the alternative (policies that actually benefit the people) will become more popular, leading to reduced unrest.

Democracy is having a bad time lately because its enemies have new weapons for use against it. If we break those weapons, it starts working again.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#239

Post by NitpickLawyer »

isodev wrote: post #223
adjfasn47573 wrote: post #218

I see most people stating that the internet as we know it could be gone because of AI. I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today? There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for…

> all sorts of people trying to exploit it for money Because they could. In AI-first web, people can't really do anything about anything - only those in control of training the handful of "big popular AI models" are the gatekeepers of all knowledge. > with less ads, more genuine information That's orthogonal to AI. Models are already being trained to favour certain products/services and they already (re)produce factu…

> only those in control of training the handful of "big popular AI models" are the gatekeepers of all knowledge.

I think that's certainly the case now, and it will be for a while, but slowly we're getting closer to that "AI personal assistant" sci-fi inspired future, where everything runs on "your" infra and gathers data / answers questions locally. You'd still need "raw" data access for that. A way to micro-pay for that would certainly help, imo.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#240

Post by eddythompson80 »

__MatrixMan__ wrote: post #231
ashdksnndck wrote: post #216

Earlier quoted context omitted.

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

It's not an ideal solution because any fixed cost solution is begging for a middle man/reseller to be introduced.

Like I pay the $5 monthly flat fee (or $500, $5k, $500k, whatever it's known fixed cost for me) for the system, turn around and resell all content for a $1 monthly flat fee.

There is a real cost to the content you're consuming with that flat-fee. So either the flat fee is more of "credit" system or it's relying on a middle man to do the oversubscribing calculation/arbitrage or whatever to balance the cost.

And no, introducing any form of rate limits or "abuse reduction" doesn't work because it's basically changing your flat-fee into a credit based system.

A credit system has advantages over pure micropayment system (in terms of mental overload. I know I charged my "internet content" card with $50 for this month. A movie on Netflix is selling for $2 tonight. Normally it's $0.5 a movie, but it's Valentines and everyone is "Netflix and Chilling" so surge charging)

Post reply on HN