Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

311–317 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#311

Post by radium3d »

Scaevolus wrote: post #131
mattlondon wrote: post #119

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps. But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point wit…

Google's "AI Overview" is massively reducing click-through rates too. At least there's a search intent unlike ChatGPT? > It used to be that for every 2 pages G scraped, you would expect 1 visitor. 6 months ago that deteriorated to 6 pages scraped to get 1 visitor. > Today the traffic ratio is: for every 18 pages Google scrapes, you get 1 visitor. What changed? AI Overviews > And that's STILL the good news. What's the…

Yeah but if Google doesn’t provide the answer I just grok it. Most of the time Google’s ai answers are wrong while grok is spot on interestingly

Re: Cloudflare to introduce pay-per-crawl for AI bots

#312

Post by dieortin »

JimDabell wrote: post #149
xela79 wrote: post #107

Earlier quoted context omitted.

>Crawling the web is not a competitive advantage for any of these AI companies, ?? it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information them not paying the content of the sites they index and read out, and not referring anybody to their sites is what will kill the web as we know it. for a website owner there is z…

> it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information My point is that you wouldn’t expect any one of them to be so much better than the others at crawling that it would give them an advantage. It’s just overhead. They all have to do it, but it doesn’t put any of them ahead. > for a website owner there is zero v…

> My point is that you wouldn’t expect any one of them to be so much better than the others at crawling that it would give them an advantage

And why not?

Re: Cloudflare to introduce pay-per-crawl for AI bots

#313

Post by BugheadTorpeda6 »

ryan_lane wrote: post #278
asim wrote: post #20

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

Except I don't want to use crypto, I don't want to accept crypto for content, I don't want to pay middlemen for using crypto. Micro payments using crypto is just a way for folks to prop up crypto currencies. It also is a dead concept, because how do we all agree on _which_ crypto to use? If I'm browsing the internet, and each site only accepts a particular shit coin, is that ok? Does everyone just use a single stable…

I don't think it would even be remotely as technically feasible or viable if it was crypto based. I don't want crypto either, but as far as I can tell, crypto is much much much less ergonomic and inconvenient than just having a tab that you pay off monthly in a normal way with a single transaction.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#314

Post by gck1 »

All that this will result in is Cloudflare starting to demand payment from humans to browse the web, eventually. It already became impossible to browse the Cloudflare-dominated web if you use VPN and/or any browser that resists fingerprinting.

Companies with a lot of money will pay up, smaller companies that depend on crawling for legitimate purposes will struggle, Cloudflare will pocket most of the profits, website owners will get pennies on the dollar and end user experience will get another blow.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#315

Post by cyounkins »

> Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.

It really seems that they did _not_ change the default, since this feature is in private beta.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#316

Post by xela79 »

JimDabell wrote: post #149
xela79 wrote: post #107

Earlier quoted context omitted.

>Crawling the web is not a competitive advantage for any of these AI companies, ?? it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information them not paying the content of the sites they index and read out, and not referring anybody to their sites is what will kill the web as we know it. for a website owner there is z…

> it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information My point is that you wouldn’t expect any one of them to be so much better than the others at crawling that it would give them an advantage. It’s just overhead. They all have to do it, but it doesn’t put any of them ahead. > for a website owner there is zero v…

> Earning money is not the only reason to have a website. Some people just want to distribute information.

yes, I just want my hosting costs covered, and that is all. Otherwise you are paying for people to steal the info you "just want to share", the info the others make a profit on... that business model is absurd.

Post reply on HN