Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

261–270 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#261

Post by tzs »

imiric wrote: post #185
tzs wrote: post #172

Earlier quoted context omitted.

You still don't need cryptocurrencies. You just need a middleman that aggregates micropayments into large enough amounts to work with non-micropayment systems. Some might object to having to get middlemen involved, but the thing is that even with cryptocurrency payments you are going to need middlemen because the web is international. If your website is directly charging crawlers to crawl and you get crawled and paid…

A middleman is not strictly required for cryptocurrencies. Regulations around them and how international transactions are taxed will depend on each country, just like anything else. These matters can be handled by lawyers and accountants as usual. While I agree that cryptocurrencies are not strictly required for this, the infrastructure already exists to support micropayments, and is well understood and trusted. What…

> Regulations around them and how international transactions are taxed will depend on each country, just like anything else. These matters can be handled by lawyers and accountants as usual.

One of my points is that quite a lot of sites don't currently do any international transactions with site visitors. They make their money selling ad space. Their transactions are with a small number of ad networks, probably in the same country.

The site's lawyers and accountants are most likely just trained in dealing with in-country transactions.

If the site start directly charging international crawlers it is then adding international transactions and will need accountants and lawyers who can deal with that.

Big sites with a lot of revenue can probably handle this fine. Smaller sites are much less likely to be able to deal with it.

There is also political risk handling it yourself because some counties are viewing AI development similarly to how they view weapon development, and I would not be surprised to find that some countries will view selling AI crawling access to certain other countries as violating sanctions.

Thus for most sites that aren't already engaged in international commerce they are probably going to want to go through a middleman to sell crawler access even if cryptocurrencies are used for the payment system.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#262

Post by ashdksnndck »

__MatrixMan__ wrote: post #231
ashdksnndck wrote: post #216

Earlier quoted context omitted.

Microtransactions are the perfect solution, if you have an economic theory that assumes near-zero transaction costs. Technology can achieve low technical costs, but the problem is the human cost of a transaction. The mental overhead of deciding whether I want to make a purchase to consume every piece of content, and whether I got ripped off, adds up, and makes microtransactions exhausting. When someone on the interne…

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

> I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated.

> The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it.

You just described bundling - that’s how YouTube Premium works. I’m not sure what the distinction you are drawing is here. Is it the existence of multiple separate bundling services? If so, I agree that creates friction, but the solution is more bundling, ie. everything should be in the same bundle.

Btw I don’t hate the fragmentation of streaming that much. The value proposition for TV/movie consumption is the best it’s ever been. For what it used to cost to buy a single season of a TV show on DVD, I now get access to watch hundreds of shows on-demand. It would be even better if all the streaming services merged together, but antitrust law will probably prevent that.

I think what most people hate more is when the specific thing they want isn’t in the bundle - ie. paying $4 to watch one movie.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#263

Post by jeroenhd »

SparkyMcUnicorn wrote: post #209
kentonv wrote: post #126

Earlier quoted context omitted.

How would an individual user use a "crawler" to navigate the web exactly? A browser that uses AI is not automatically a "crawler"... a "crawler" is something that mass harvests entire web sites to store for later processing...

How can you tell the difference, in a way that can't be spoofed? This is a genuine question, since I see you work at CF. I'm very curious what the distinction will be between a user and a crawler. Is trust involved, making spoofing a non-issue?

Not OP of course, but I think there's a clear way forward.

An LLM accessibility browser is a bot, so bot detection sounds like the wrong approach to me. What's more important than bot detection is "actual real user" detection, of which bot detection is only part.

If the control software runs on a user's local device, things like TPMs can offer a device-bound signature for remote attestation. Virtual TPMs don't have root certificates signed by TPM/CPU makers, so they're not useful for building trust. A CPU shared between hundreds of other VMs somewhere in a cloud will not be providing unique TPM verification so AI scrapers will have to switch their scraping to having botnets do the work rather than just using them as proxies, and even then they can't get away with hacked routers (that lack TPMs).

There's a huge downside to this, of course, and that's basically handing control over who gets to use the internet to a few TPM companies that can lock you out whenever they please. If there's any way to tie this remote attestation system to you as a person, this puts tremendous power in the hands of the US government (see what happened to the ICC judge investigating the genocide over in Gaza) as they can force American companies to banish you.

I don't think the internet should develop in this direction, but with CAPTCHA failing to block bots and with AI scrapers ruining the internet, I don't see things going any other way.

Now that Cloudflare is putting a monetary value to bypassing its blocks for shitty AI scrapers, you can bet that there will be an industry of underpaid IT workers figuring out how to bypass CF's bot detection for a competitive market rate.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#264

Post by ashdksnndck »

eddythompson80 wrote: post #240
__MatrixMan__ wrote: post #231

Earlier quoted context omitted.

I disagree, bundling is the problem. That strategy created the fragmented landscape that we now see in streaming video, which is pretty much universally hated. The ideal solution would involve a flat rate which I pay monthly, and at the end of the month that money goes towards the content that I consumed during that month. If I only read a single blog, they get all of it. Then we build a culture around preferring to…

It's not an ideal solution because any fixed cost solution is begging for a middle man/reseller to be introduced. Like I pay the $5 monthly flat fee (or $500, $5k, $500k, whatever it's known fixed cost for me) for the system, turn around and resell all content for a $1 monthly flat fee. There is a real cost to the content you're consuming with that flat-fee. So either the flat fee is more of "credit" system or it's r…

I think the costs of serving digital content are so low that you don’t need to rely on oversubscription. The user can stream 24/7 and you would still make money (assuming you got the working IP payment model like YouTube does that divide up individual user revenue proportional to that user’s watch time - I think Spotify has a problem here). The only “anti abuse” you need is to enforce that the user only streams one thing at a time.

The problem with the credit system is that the user won’t like that they have to pay extra for the good stuff, the feeling of watching worse stuff to save money etc.

Given the marginal cost of distributing the good stuff is the same as the bad stuff, why make the customer feel bad about watching by adding an incremental cost? Just let it rip. If you have a lot of good stuff, customers will be willing to pay more for the bundle. Once they’re in the bundle, let them watch exactly what they want.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#265

Post by PhilippGille »

asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

It's not a new idea. For example I created https://github.com/philippgille/ln-paywall (also using 402 status code) in 2018.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#266

Post by kentonv »

jeroenhd wrote: post #263
SparkyMcUnicorn wrote: post #209

Earlier quoted context omitted.

How can you tell the difference, in a way that can't be spoofed? This is a genuine question, since I see you work at CF. I'm very curious what the distinction will be between a user and a crawler. Is trust involved, making spoofing a non-issue?

Not OP of course, but I think there's a clear way forward. An LLM accessibility browser is a bot, so bot detection sounds like the wrong approach to me. What's more important than bot detection is "actual real user" detection, of which bot detection is only part. If the control software runs on a user's local device, things like TPMs can offer a device-bound signature for remote attestation. Virtual TPMs don't have r…

> An LLM accessibility browser is a bot

I don't agree with this. A browser, operated by a human user, is not a bot. Adding LLM-powered accessibility features to a browser does not make it a bot.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#267

Post by jahewson »

imiric wrote: post #46
asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Sorry but the sociopolitical unrest of the last decade has been caused by the problems we have created not by people talking about it.

Money need not be involved, just look at how corrupt and biased Wikipedia has become.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#268

Post by wongarsu »

giantrobot wrote: post #170
wongarsu wrote: post #147

Earlier quoted context omitted.

As a Startup I absolutely want to get crawled. If people ask ChatGPT "Who is $CompanyName" I want it to give a good answer that reflects our main USPs and talking points. A lot of classic SEO content also makes great AI fodder. When I ask AI tools to search the web to give me a pro/con list of tools for a specific task the sources often end up being articles like "top 10 tools for X" written by one of the companies o…

> A lot of classic SEO content also makes great AI fodder. Huh? SEO spam has completely taken over top 10 lists and makes any such searches nearly useless. This has been the case for at least a decade. That entire market is 1000% about getting clicks. Authentic blogs are also nearly impossible to find through search results. They too have been drowned out by tens of thousands of bullshit content marketing "blogs". Be…

If I search for "best mlops tool" (granted, generic and uninspired query) on google, in the first 5 results I get three blogs of course platforms, one reddit thread, one github "awesome list" and 5 "top X" lists made by mlops tools, usually with that exact tool as the number one recommendation.

Of those 10 results, only one is ad-financed (reddit). And at least the five mlops tools won't mind being crawled and regurgitated algorithmically. If an AI uses their biased list to form opinions and recommendations that's exactly what they want

Re: Cloudflare to introduce pay-per-crawl for AI bots

#269

Post by 1659447091 »

blackjack_ wrote: post #220
squigz wrote: post #196

Earlier quoted context omitted.

Okay, but my question remains - why? What is the connection between those things and advertising? It seems way more likely to me that they would simply adapt, as they always have.

Sure, I can explain. Basically, adtech is the backbone of the attention economy where more clicks = more revenue. So the incentives are to always say the most inflammatory clickbait you can, to incentivize profits. Sensible and boring stable takes and agreement will always be stifled to promote outrage, beefs, and clickbait to maximize revenue. To generalize; stability in any general field like politics or journalism…

So what you are saying is: "incentives are to always say the most inflammatory clickbait you can, to incentivize profits"

Therefore in this new setup where people pay sites and not other companies, if I want the most micro-payments coming to my site, I need to say the most inflammatory clickbait things I can? All this does is shift the who pays, and then of course because I want the most money for my site, I will also take a company's ad money and do tiers with the micro payments now. At no point does that change the content people want. Sure, maybe I won't get your micro payment, but that okay because now I have a new scheme that gives me advertisers money and readers money. Now if I find a way to exploit the FOMO of this new gated setup, I win even more and my content doesn't change.

These takes always find ways to blame providers and never hold to account the responsibility of consumers. Maybe we focus on the people and why they want the clickbait so bad? I don't have an answer to that, but that's probably because the "solutions" all want to focus on companies giving people exactly what they want instead of helping people become aware of addictive clickbait rage behaviour and patterns.

Fox tv station is not the only one that is broadcast into peoples homes, PBS is an option too. And on cable for news, my grandmother had C-SPAN on tv non-stop. Talk about some boring stable tv. (Which I am sure is still an option if thats what people want)

Re: Cloudflare to introduce pay-per-crawl for AI bots

#270

Post by 1dom »

thelastkek wrote: post #173
1dom wrote: post #98

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl. On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better. This…

I almost guarantee Cloudflare will give this feature away for free for all tiers (Free plan included). I doubt they will make a penny off this. If you knew the company history and culture, youd foresee that too.

I do know the company history, I've been a customer for well over a decade. As far as big tech goes, they've got fewer skeletons than most, but you're naive if you think they're not making a penny off this in any way.
Post reply on HN