Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

191–200 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#191
post #46
post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

> This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity.

People with content will still want to maximize their money. You'll get all the same bullshit dark patterns on sites supported by microtransactions as you will ad supported. Stories will be split up into multiple individual pages, each requiring a microtransaction. Even getting past a landing page will require multiple click throughs each with another transaction. There will also be nothing preventing sites from bait and switch schemes where the link exposed to crawlers doesn't contain the expected content.

Without extensive support for micro-refunds and micro-customer service and micro-consumer protections, microtransactions on the web will most likely lead to more abusive bullshit. Automated integrations with browsers will be exploited.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#192
post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

As someone who has actually built working micro payments systems, this was of interest. Worth noting though that it's really just "document-ware" -- there's no code there[1], and their proposed protocol doesn't look like it was thought through to the point where it has all the pieces that would be needed. [1] E.g. this file is empty: https://github.com/coinbase/x402/blob/main/package.json

> Worth noting though that it's really just "document-ware" -- there's no code there

That's not true. That project is a monorepo, with reference client and middleware implementations in TypeScript, Python, Java, and Go. See their respective subdirectories. There's also a 3rd-party Rust implementation[1].

You can also try out their demo at [2]. So it's a fully working project.

[1]: https://github.com/x402-rs/x402-rs

[2]: https://www.x402.org/

Re: Cloudflare to introduce pay-per-crawl for AI bots

#193

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Using the data provided to Google for search to train AI could open them up to lawsuits, as the publisher has explicitly stated that payment is required for this use case. They might win the class action, but would they bother risking it?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#194

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

[deleted]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#195

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So how will Cloudflare detect bots and get them to pay? And how many humans and legitimate…

[deleted]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#196
post #189
post #121

Earlier quoted context omitted.

Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?

Where did I say that all of those things would stop? What I said is that adtech systems are also used for it. So if they were to disappear overnight, a _proportion_ of those activities, and a pretty large one I reckon, would also disappear.

Okay, but my question remains - why? What is the connection between those things and advertising?

It seems way more likely to me that they would simply adapt, as they always have.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#197
post #142
post #63

Earlier quoted context omitted.

crypto seems like a massive waste for what can just be a regular transaction

Much cheaper than using a credit card processor.

Not cheaper than using one of the instant net settlement services like FedNow (UPI in India, PromptPay in Thailand, PayID in Australia)

Re: Cloudflare to introduce pay-per-crawl for AI bots

#198
post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

> This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. People with content will still want to maximize their money. You'll get all the same bullshit dark patterns on sites supported by microtransactions as you will ad supported. Stories will be split up into multiple individual pages, each re…

Maybe. But at least transactions could be performed directly between consumers and publishers, and there wouldn't be incentives for companies to violate privacy laws and exploit user data.

Of course, we would need to figure out solutions to a bunch of problems adtech companies have had decades to do, but micropayments would be the first step in the right direction. A larger hurdle would be educating users into paying for content, and what "free" has meant thus far, so that they could make an informed decision. And even then I expect that many people would prefer paying with their attention and data instead. But giving the option for currency payment with _zero_ ads is something that can be forced by regulation, which I hope happens one day.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#199

Earlier quoted context omitted.

> Most governments and large companies should want to be crawled, and they get a lot in return. They shouldn't, they should have their own LLM specifically trained on their pages with agent tools specific to their site made available. It's the only way to be sure that the answers given are not garbage. Citizens could be lost on how to use federal or state websites if the answers returned by Google are wrong or outdat…

This is ignoring how people use things.

No, it's taking back control of what tools can be used to achieve a specific goal.

If Google can't guarantee a good user experience but also correctness of the informations returned by their LLM than a ministry shouldn't stand for this and setup their own tools.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#200
post #196
post #189

Earlier quoted context omitted.

Where did I say that all of those things would stop? What I said is that adtech systems are also used for it. So if they were to disappear overnight, a _proportion_ of those activities, and a pretty large one I reckon, would also disappear.

Okay, but my question remains - why? What is the connection between those things and advertising? It seems way more likely to me that they would simply adapt, as they always have.

The connection is that those wishing to influence public opinion can do so by running ad campaigns that target precisely the demographic they wish to manipulate. Adtech doesn't care whether you're promoting products or ideas. This connection should be obvious after the Cambridge Analytica leak.

Social media and any media platform also enables the spreading of propaganda, but it's not as systematic as the tools built for advertising.

Post reply on HN