Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

141–150 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#141

Post by rickdeckard »

dhx wrote: post #112
OtherShrezzing wrote: post #38

Earlier quoted context omitted.

I think you're looking at the wrong side of the market for the incentive structures here. Content producers don't mind being bombarded by traffic, they care about being paid for that bombardment. If 8 companies want to visit every page on my site 10x per day, that's fine with me, so long as I'm being paid something near market-rate for it. For the 8 companies, they're then incentivised to collaborate on a unified cra…

It depends on the content producer. I would argue the best resourced content producers (governments and large companies) are incentivised to give AI bots as much curated content as possible that is favourable to their branding and objectives. Even if it's just "soft influence" such as the French government feeding AI bots an overwhelming number of articles about how the Eiffel Tower is the most spectacular tourist at…

> It depends on the content producer. I would argue the best resourced content producers (governments and large companies) are incentivised to give AI bots as much curated content as possible that is favourable to their branding and objectives.

Yeah, if the content being processed is NOT the product being sold by the creator.

> [..] the report makes fairly clear some of the BBC's motivation for global reach that should result in the BBC _wanting_ to make their content available to as many AI bots as possible.

What kind of monetization model would this be for BBC?

"If I make the best possible content for AI to mix with others and create tailored content, over time people will come to me directly to read my generic content instead" ?

It reminds me of "IE6, the number one browser to download other browsers", but worse

Re: Cloudflare to introduce pay-per-crawl for AI bots

#142

Post by bo1024 »

hhh wrote: post #63
imiric wrote: post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

crypto seems like a massive waste for what can just be a regular transaction

Much cheaper than using a credit card processor.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#143

Post by vasilzhigilei »

DocTomoe wrote: post #132
vasilzhigilei wrote: post #123

Man, HN is sleeping on this right now. This is huge. 20% of the web is behind Cloudflare. What if this was extended to all customers, even the millions of free ones? Would be really amazing to get paid to use Cloudflare as a blog owner, for example

The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…

I worked at Cloudflare for 3 years until very recently, and it's simply not the culture to behave in the way that you are describing.

There exists a strong sense of doing the thing that is healthiest for the Internet over what is the most profit-extractive, even when the cost may be high to do so or incentives great to choose otherwise. This is true for work I've been involved with as well as seeing the decisions made by other teams.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#145

Post by ipaddr »

vbezhenar wrote: post #103
jimbohn wrote: post #66

Earlier quoted context omitted.

>Google with get it for free because Google Search What if the second step is that Google pays the page it visits? By enabling a crawler fee per page, news websites could make some articles uncrawlable unless a huge fee is paid. Just thinking aloud, but I could easily see a protocol stating pricing by different kinds of "licensing" e.g. "internal usage", "redistribution" (what google news did/does?), "LLM training",…

The question is: who has the leverage? If some small news website denies Google Bot crawling, it'll disappear from Google and essentially it'll disappear from the Internet. People do a great lengths to appease the Google Crawler. If some huge news website demands fees from Google, it might work, I guess. But I'm not sure that it would work even for BBC or CNN.

If its a smaller news site they have already de-ranked them, and used their content for AI answers

Re: Cloudflare to introduce pay-per-crawl for AI bots

#146

Post by 4b11b4 »

squigz wrote: post #121
imiric wrote: post #46

Earlier quoted context omitted.

This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…

Even if advertising were to disappear over night, why do you think that would stop the spread of propaganda, corruption of democratic processes, and social unrest? I don't really see a connection between the two?

It's more like the tech that allows middlemen to insert into everything and be very hyper personalized/targeted

Re: Cloudflare to introduce pay-per-crawl for AI bots

#147

Post by wongarsu »

mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

As a Startup I absolutely want to get crawled. If people ask ChatGPT "Who is $CompanyName" I want it to give a good answer that reflects our main USPs and talking points.

A lot of classic SEO content also makes great AI fodder. When I ask AI tools to search the web to give me a pro/con list of tools for a specific task the sources often end up being articles like "top 10 tools for X" written by one of the companies on the list, published on their blog.

Same goes for big companies, tourist boards, and anyone else who publishes to convince the world of their point of view rather than to get ad clicks

Re: Cloudflare to introduce pay-per-crawl for AI bots

#148

Post by Workaccount2 »

Scaevolus wrote: post #131
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Google's "AI Overview" is massively reducing click-through rates too. At least there's a search intent unlike ChatGPT? > It used to be that for every 2 pages G scraped, you would expect 1 visitor. 6 months ago that deteriorated to 6 pages scraped to get 1 visitor. > Today the traffic ratio is: for every 18 pages Google scrapes, you get 1 visitor. What changed? AI Overviews > And that's STILL the good news. What's the…

Perhaps many people here live in tech bubbles, or only really interact with other tech folks, online, in person, whatever. People in tech are relatively grounded about LLMs. Relatively being key here.

On the ground in normal people society, I have seen that people just treat AI as the new fountain of answers and aren't even aware of LLM's tendency to just confidently state whatever it conjures up. In my non-tech day to day life, I have yet to see someone not immediately reference AI overview when searching something. It gets a lot of hostility in tech circles, but in real life? People seem to love it.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#149

Post by JimDabell »

xela79 wrote: post #107
JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

>Crawling the web is not a competitive advantage for any of these AI companies, ?? it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information them not paying the content of the sites they index and read out, and not referring anybody to their sites is what will kill the web as we know it. for a website owner there is z…

> it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information

My point is that you wouldn’t expect any one of them to be so much better than the others at crawling that it would give them an advantage. It’s just overhead. They all have to do it, but it doesn’t put any of them ahead.

> for a website owner there is zero value of having their content indexed by AI bots. Zilch.

Earning money is not the only reason to have a website. Some people just want to distribute information.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#150

Post by squigz »

dhx wrote: post #137
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

> But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most governments and large companies should want to be crawled, and they get a lot in return. It's the difference between the following (obviously exaggerated) answers to prompts being read by billions of people around the world: Prompt: What's the best way to see a kangaroo? Response…

I'd be unsatisfied with both of those answers. 1 is an advertisement, and the other is pretty long-winded - and of course, I have no way of knowing whether either are correct
Post reply on HN