Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

151–160 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#151

Post by Workaccount2 »

DocTomoe wrote: post #132
vasilzhigilei wrote: post #123

Man, HN is sleeping on this right now. This is huge. 20% of the web is behind Cloudflare. What if this was extended to all customers, even the millions of free ones? Would be really amazing to get paid to use Cloudflare as a blog owner, for example

The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…

It's worse than that, it strongly incentivizes creating agents that spin up blogs, fill them with LLM vomit, and then enable "pay-for-training".

It's basically creating a "get paid to spam the internet with anything" system.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#152

Post by squigz »

Workaccount2 wrote: post #148
Scaevolus wrote: post #131

Earlier quoted context omitted.

Google's "AI Overview" is massively reducing click-through rates too. At least there's a search intent unlike ChatGPT? > It used to be that for every 2 pages G scraped, you would expect 1 visitor. 6 months ago that deteriorated to 6 pages scraped to get 1 visitor. > Today the traffic ratio is: for every 18 pages Google scrapes, you get 1 visitor. What changed? AI Overviews > And that's STILL the good news. What's the…

Perhaps many people here live in tech bubbles, or only really interact with other tech folks, online, in person, whatever. People in tech are relatively grounded about LLMs. Relatively being key here. On the ground in normal people society, I have seen that people just treat AI as the new fountain of answers and aren't even aware of LLM's tendency to just confidently state whatever it conjures up. In my non-tech day…

And this is why we can't just rely on awareness of these issues - we need to also hold companies accountable for false information.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#153

Post by wongarsu »

miohtama wrote: post #140
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Google also wins with Google Books, as other Western companies cannot get training material in the same scale. Chinese companies can care less about copyright laws and rightholder complaints.

Google's advantage is mostly in historical books. Google Books has a great collection going back to the 1500s.

For modern works anyone can just add Z-Library and Anna's Archive. Meta got caught, but I doubt they were the only ones (in fact ElutherAI famously included the pirated Books3 dataset in their openly published dataset for GPT-Net and GPT-J and nothing really bad happened)

Re: Cloudflare to introduce pay-per-crawl for AI bots

#156

Post by seanw444 »

vasilzhigilei wrote: post #143
DocTomoe wrote: post #132

Earlier quoted context omitted.

The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…

I worked at Cloudflare for 3 years until very recently, and it's simply not the culture to behave in the way that you are describing. There exists a strong sense of doing the thing that is healthiest for the Internet over what is the most profit-extractive, even when the cost may be high to do so or incentives great to choose otherwise. This is true for work I've been involved with as well as seeing the decisions mad…

Unfortunately, even if it is as you describe, human nature is such that it will not stay that way forever. Likely not even for long.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#157

Post by focusedone »

vasilzhigilei wrote: post #143
DocTomoe wrote: post #132

Earlier quoted context omitted.

The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…

I worked at Cloudflare for 3 years until very recently, and it's simply not the culture to behave in the way that you are describing. There exists a strong sense of doing the thing that is healthiest for the Internet over what is the most profit-extractive, even when the cost may be high to do so or incentives great to choose otherwise. This is true for work I've been involved with as well as seeing the decisions mad…

That's the impression I get from Cloudflare - it seems like a group of highly skilled people attempting to solve real problems for the benefit of the web as a whole. As both a paid business user and a free user for home projects, I deeply appreciate what they've accomplished and how generously they allow unpaid users to benefit from their work.

I worry about what happens someday when leadership changes and the priority becomes value extraction rather than creation, if that makes sense. We've seen it so many times with so many other tech companies, it's difficult to believe it won't happen to Cloudflare at some point.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#158

Post by vevoe »

Workaccount2 wrote: post #151
DocTomoe wrote: post #132

Earlier quoted context omitted.

The cynic in me says we'll be seeing articles about blog owners getting fractions of a tenth of a penny while Cloudflare pockets most of the revenue. And of course it will eventually be rolled out for everyone, meaning there will be a Cloudflare-Net (where you only can read if you give Cloudflare your credit card number), and then successively more competing infrastructure services (Akamai, AWS, ... meaning we get in…

It's worse than that, it strongly incentivizes creating agents that spin up blogs, fill them with LLM vomit, and then enable "pay-for-training". It's basically creating a "get paid to spam the internet with anything" system.

tbf I think that's already been happening for a while now

Re: Cloudflare to introduce pay-per-crawl for AI bots

#159

Post by LunaSea »

dhx wrote: post #137
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

> But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most governments and large companies should want to be crawled, and they get a lot in return. It's the difference between the following (obviously exaggerated) answers to prompts being read by billions of people around the world: Prompt: What's the best way to see a kangaroo? Response…

> Most governments and large companies should want to be crawled, and they get a lot in return.

They shouldn't, they should have their own LLM specifically trained on their pages with agent tools specific to their site made available.

It's the only way to be sure that the answers given are not garbage.

Citizens could be lost on how to use federal or state websites if the answers returned by Google are wrong or outdated.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#160

Post by mmarian »

mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

I'm not sure it'll work though. Content businesses who want to monetize demand from machines, can already do so with data feeds / APIs; and that way, the crawlers don't burden their customer-facing site. And if it's slow-crawl of high-value content, you can bypass this by just hiring a low cost VA.

Is there anything I'm missing?

Post reply on HN