Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

161–170 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#161

Post by petesergeant »

boplicity wrote: post #138
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Not sure how Google is winning AI, at least from the sophisticated consumer's perspective. Their AI overviews are often comically wrong. Sure, they may have Good APIs for their AI, and good technical quality for their AIs, but for the general user, their most common AI presentation is woefully bad.

> Not sure how Google is winning AI

I don't especially think they are, but if I was trying to argue it, I'd note that Gemini is a very, very capable model, and Google are very well-placed to sell inference to existing customers in a way I'm less sure that OpenAI and Anthropic are.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#162

Post by ddingus »

Workaccount2 wrote: post #148
Scaevolus wrote: post #131

Earlier quoted context omitted.

Google's "AI Overview" is massively reducing click-through rates too. At least there's a search intent unlike ChatGPT? > It used to be that for every 2 pages G scraped, you would expect 1 visitor. 6 months ago that deteriorated to 6 pages scraped to get 1 visitor. > Today the traffic ratio is: for every 18 pages Google scrapes, you get 1 visitor. What changed? AI Overviews > And that's STILL the good news. What's the…

Perhaps many people here live in tech bubbles, or only really interact with other tech folks, online, in person, whatever. People in tech are relatively grounded about LLMs. Relatively being key here. On the ground in normal people society, I have seen that people just treat AI as the new fountain of answers and aren't even aware of LLM's tendency to just confidently state whatever it conjures up. In my non-tech day…

They do love it. I have been, nicely and as helpfully as I can, educating people on the nature of LLM tools.

I personally have little hostility toward the AI search results. Most of the time, the feature nails my quick search queries. Those are usually on something I need a detail filled in due to forgetting said detail, or a slightly different use case where I am already familiar enough to catch gaffes.

Anything else and I typically ignore it and do my usual search elsewhere, or fast scroll down to the worthy site links.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#164

Post by hubraumhugo »

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space).

So how will Cloudflare detect bots and get them to pay? And how many humans and legitimate bots will get blocked as a side effect? We're somehow still stuck with CAPTCHAs, a 25 years old concept that wastes millions of human hours and billions in infra costs [0].

How can we enable beneficial automation while protecting against abusive AI crawlers?

[0] https://arxiv.org/abs/2311.10911

Re: Cloudflare to introduce pay-per-crawl for AI bots

#165

Post by danaris »

gessha wrote: post #127
danaris wrote: post #82

Earlier quoted context omitted.

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in. "Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to t…

The commons are not destined to become a tragedy and they can become a long-term resource everyone can enjoy[1]. You need clear boundaries, reliable monitoring of shared resource, reasonable balance between costs and benefits, etc. > I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to this company Or weasel out of being a non-profit. [1] https://aeon.co/essays/t…

Hm. I hadn't understood the Tragedy of the Commons to be an inevitability, merely a phenomenon—something that does happen sometimes, not something that must happen all the time.

And unfortunately, in our current culture, at least in the US, it's much more likely than not when the circumstances allow it. We will need generations' worth of work firmly demonstrating that things can be better for everyone when we all agree to share in things equally, rather than allowing individuals to take what's meant for everyone.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#166

Post by BenjiWiebe »

squigz wrote: post #155
freeone3000 wrote: post #135

Earlier quoted context omitted.

Honestly preferable to the insane amounts of paywalls and advertising

This is a paywall.

I'd rather pay 5c for one article than subscribe for $10/yr to view one article. Still a paywall, but less annoying.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#167

Post by [deleted] »

dboreham wrote: post #130
asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

As someone who has actually built working micro payments systems, this was of interest. Worth noting though that it's really just "document-ware" -- there's no code there[1], and their proposed protocol doesn't look like it was thought through to the point where it has all the pieces that would be needed. [1] E.g. this file is empty: https://github.com/coinbase/x402/blob/main/package.json

[deleted]

Re: Cloudflare to introduce pay-per-crawl for AI bots

#168

Post by J_Shelby_J »

asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

How do you handle KYC?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#169

Post by dhx »

squigz wrote: post #150
dhx wrote: post #137

Earlier quoted context omitted.

> But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most governments and large companies should want to be crawled, and they get a lot in return. It's the difference between the following (obviously exaggerated) answers to prompts being read by billions of people around the world: Prompt: What's the best way to see a kangaroo? Response…

I'd be unsatisfied with both of those answers. 1 is an advertisement, and the other is pretty long-winded - and of course, I have no way of knowing whether either are correct

Try a subjective prompt such as "which country has the most advanced car manufacturing industry" and you'll get responses with common subjective biases such as:

- Reliability: Japan

- Luxury: Germany

- Cost, EV batteries, manufacturing scale: China

- Software: USA

(similar output for both deepseek-r1-0528 and gemini-2.5-pro tested)

These LLM biases are worth something to the countries (and companies within) that are part of the automotive industry. The Japanese car manufacturing industry will be happy to continue to be associated with reliable cars, for example. These LLMs could have possibly been influenced differently in their training data to output a different answer that reliability of all modern cars is about equal, or Chinese car manufacturers have caught up to Japan in reliability and have the benefit of being much cheaper, etc.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#170

Post by giantrobot »

wongarsu wrote: post #147
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

As a Startup I absolutely want to get crawled. If people ask ChatGPT "Who is $CompanyName" I want it to give a good answer that reflects our main USPs and talking points. A lot of classic SEO content also makes great AI fodder. When I ask AI tools to search the web to give me a pro/con list of tools for a specific task the sources often end up being articles like "top 10 tools for X" written by one of the companies o…

> A lot of classic SEO content also makes great AI fodder.

Huh? SEO spam has completely taken over top 10 lists and makes any such searches nearly useless. This has been the case for at least a decade. That entire market is 1000% about getting clicks. Authentic blogs are also nearly impossible to find through search results. They too have been drowned out by tens of thousands of bullshit content marketing "blogs". Before they were AI slop they were Fiverr slop.

Post reply on HN