Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

171–180 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#171

Post by davemel37 »

Workaccount2 wrote: post #148
Scaevolus wrote: post #131

Earlier quoted context omitted.

Google's "AI Overview" is massively reducing click-through rates too. At least there's a search intent unlike ChatGPT? > It used to be that for every 2 pages G scraped, you would expect 1 visitor. 6 months ago that deteriorated to 6 pages scraped to get 1 visitor. > Today the traffic ratio is: for every 18 pages Google scrapes, you get 1 visitor. What changed? AI Overviews > And that's STILL the good news. What's the…

Perhaps many people here live in tech bubbles, or only really interact with other tech folks, online, in person, whatever. People in tech are relatively grounded about LLMs. Relatively being key here. On the ground in normal people society, I have seen that people just treat AI as the new fountain of answers and aren't even aware of LLM's tendency to just confidently state whatever it conjures up. In my non-tech day…

I mentioned hallucinations last week on a call with 2 seasoned marketers and both thought I invented the term on the spot.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#172

Post by tzs »

dboreham wrote: post #128
gessha wrote: post #120

Earlier quoted context omitted.

But why exactly does it have to be on an append-only ledger where transactions are processed/validated for a fee? Why can’t it be a more conventional transaction processor like VISA on top of the banking system?

Because Visa doesn't (hasn't) wanted to do microtransactions, ever, since the beginning of the internet.

You still don't need cryptocurrencies.

You just need a middleman that aggregates micropayments into large enough amounts to work with non-micropayment systems.

Some might object to having to get middlemen involved, but the thing is that even with cryptocurrency payments you are going to need middlemen because the web is international.

If your website is directly charging crawlers to crawl and you get crawled and paid by any crawler from another country, congratulations! You are now engaged directly in international trade and have a whole slew of regulations to deal with, probably from both your country and the country the crawler is from.

If you go through a middleman you can structure things so it is the middleman that is buying crawler access from you. Pick a middleman in your country (or anywhere in the EU of you are in the EU) and most of your regulatory headaches go away.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#173

Post by thelastkek »

1dom wrote: post #98

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl. On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better. This…

I almost guarantee Cloudflare will give this feature away for free for all tiers (Free plan included). I doubt they will make a penny off this. If you knew the company history and culture, youd foresee that too.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#174

Post by xboxnolifes »

LunaSea wrote: post #159
dhx wrote: post #137

Earlier quoted context omitted.

> But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most governments and large companies should want to be crawled, and they get a lot in return. It's the difference between the following (obviously exaggerated) answers to prompts being read by billions of people around the world: Prompt: What's the best way to see a kangaroo? Response…

> Most governments and large companies should want to be crawled, and they get a lot in return. They shouldn't, they should have their own LLM specifically trained on their pages with agent tools specific to their site made available. It's the only way to be sure that the answers given are not garbage. Citizens could be lost on how to use federal or state websites if the answers returned by Google are wrong or outdat…

This is ignoring how people use things.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#175

Post by imiric »

mattlondon wrote: post #124
kevlened wrote: post #65

Earlier quoted context omitted.

Google had an experiment called Google Contributor where you could buy all your own ads. This effectively had the experience you're describing (prepay and get no ads until it runs out). They tried it twice, so someone wanted it to work. I was always curious why they shuttered it.

Probably because people didn't want to pay. It's easy and cheap to say "I'd pay X to access website without ads!" ...but when it came to it and people had that option, essentially no-one did it.

This is because most of the time paying is not an option. And even when it is, there is a lot of friction to actually do that, even with streamlined payment services such as Stripe. The advertising business model and technology that powers it is so well established that a "free" service is much easier to manage for publishers and to access for consumers.

There's also the psychological aspect. People are used to advertising in every other form of media, so seeing it online is acceptable. People expect online services to be "free", and few really understand the business transaction they're a part of, or the repercussions of it. Even when they do, many are willing to make that transaction because they value the service more than what they're giving up to access it, and they have no other choice.

So it ultimately boils down to offering the choice to pay with currency, and making it frictionless for both consumers and publishers to use. And educating consumers about the real cost of advertising.

The unfortunate reality is that advertising has become so profitable that in order for the payment system to work, companies would have to price their services higher than any consumer would be willing to pay for them. Or they would have to settle for lower profits, which no company would actually do. This is why you see that even when a service has a payment option, they still inevitably choose to _also_ show ads. Advertising money is simply irresistible to most people, and few have the moral backbone to resist it.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#177

Post by glenstein »

dhx wrote: post #169
squigz wrote: post #150

Earlier quoted context omitted.

I'd be unsatisfied with both of those answers. 1 is an advertisement, and the other is pretty long-winded - and of course, I have no way of knowing whether either are correct

Try a subjective prompt such as "which country has the most advanced car manufacturing industry" and you'll get responses with common subjective biases such as: - Reliability: Japan - Luxury: Germany - Cost, EV batteries, manufacturing scale: China - Software: USA (similar output for both deepseek-r1-0528 and gemini-2.5-pro tested) These LLM biases are worth something to the countries (and companies within) that are…

Those companies can want that all they want, meanwhile the developers of LLMs themselves can choose or not choose to reflect that in their training or to monetize their training.

You're absolutely right that there's an interest in affecting the output, but my hope is the design of models is not influenced by this, or that we can know enough about how models are designed to prefer ones that are not nudged in this way.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#178

Post by nerdix »

freeone3000 wrote: post #135
skenderbeu wrote: post #86

How long before we get pay per browse and the internet is 6ft under?

Honestly preferable to the insane amounts of paywalls and advertising

That won't end ads.

Just like paid cable subscriptions didn't end TV ads. Or how ads are slowly creeping into the various streaming platforms with "ad supported tiers".

Re: Cloudflare to introduce pay-per-crawl for AI bots

#179

Post by gessha »

kevlened wrote: post #65
OtherShrezzing wrote: post #44

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

Google had an experiment called Google Contributor where you could buy all your own ads. This effectively had the experience you're describing (prepay and get no ads until it runs out). They tried it twice, so someone wanted it to work. I was always curious why they shuttered it.

Does Google need a reason to shutter something?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#180

Post by sofixa »

nottorp wrote: post #96
mdrzn wrote: post #55

Earlier quoted context omitted.

What if I save the page locally on my machine? Or archive it?

What if my phone rings and I forget to close the page? Or leave it open to read it later? Plus as one of the parent comments said, I am not paying before I get an idea what I'm paying for.

> Plus as one of the parent comments said, I am not paying before I get an idea what I'm paying for.

The logic was, you aren't paying to a website. You're paying to a broker that distributes how much you've paid to all websites you've visited that have opted in, prorated based on how much time you spent on them.

Post reply on HN