Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

101–110 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#101

Post by vbezhenar »

1dom wrote: post #98

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl. On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better. This…

> This should be implemented as a web protocol with crypto though so anyone can charge bots without having to be Cloudflare fronted. Not really a fanboi of 99% of crypto stuff, but IMO, a purely technical, open and decentralised solution to this sort of problem was the crypto dream.

It's not just about payment. It's about refusing to serve content to bots, unless they paid. It might be hard to implement without Cloudflare, when bot developers specifically target your website.

The whole point of Cloudflare is to let them decide whether it's bot or user that hits your website. It is complicated task.

Unless you want to force all users to pay, both humans and bots.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#102

Post by Intralexical »

joosters wrote: post #90
danaris wrote: post #82

Earlier quoted context omitted.

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in. "Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to t…

You can tell the difference between the two by checking if the Evil bit is set in the corresponding IP packet - RFC 3514 already standardised this.

If that doesn't work, you can also add rate limiting by enforcing compliance with RFC 1149.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#103

Post by vbezhenar »

jimbohn wrote: post #66
Toritori12 wrote: post #3

Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.

>Google with get it for free because Google Search What if the second step is that Google pays the page it visits? By enabling a crawler fee per page, news websites could make some articles uncrawlable unless a huge fee is paid. Just thinking aloud, but I could easily see a protocol stating pricing by different kinds of "licensing" e.g. "internal usage", "redistribution" (what google news did/does?), "LLM training",…

The question is: who has the leverage?

If some small news website denies Google Bot crawling, it'll disappear from Google and essentially it'll disappear from the Internet. People do a great lengths to appease the Google Crawler.

If some huge news website demands fees from Google, it might work, I guess. But I'm not sure that it would work even for BBC or CNN.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#104

Post by imglorp »

1dom wrote: post #98

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl. On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better. This…

Yes, right, it should be an open protocol so any CDN or content provider can use it the same way. Hopefully it becomes a part of popular web servers so little guys can play along without a CDN.

It needn't be crypto, but would be convenient. Lacking that, it would need some unforgeable presentation of identity that could be connected to a bank account.

I shed not one tear for the crawlers - they had their chance to respect robots.txt on the honor system. Now we force them.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#105

Post by jimbohn »

vbezhenar wrote: post #103
jimbohn wrote: post #66

Earlier quoted context omitted.

>Google with get it for free because Google Search What if the second step is that Google pays the page it visits? By enabling a crawler fee per page, news websites could make some articles uncrawlable unless a huge fee is paid. Just thinking aloud, but I could easily see a protocol stating pricing by different kinds of "licensing" e.g. "internal usage", "redistribution" (what google news did/does?), "LLM training",…

The question is: who has the leverage? If some small news website denies Google Bot crawling, it'll disappear from Google and essentially it'll disappear from the Internet. People do a great lengths to appease the Google Crawler. If some huge news website demands fees from Google, it might work, I guess. But I'm not sure that it would work even for BBC or CNN.

I agree about the leverage and small website reasoning, definitely some game-theory related thinking is needed to get something like this right. But it does feel like this enables the "unionization" of websites against scraping giants, google is in an especially interesting position because, as you mentioned, could blackmail you into scraping in exchange for indexing.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#106

Post by carlosjobim »

OtherShrezzing wrote: post #44

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

I've been suggesting exactly this for some time. A button in the browser to pay the asked value to view a page, and if the page is free to view the button instead turns into a donation button for voluntary donations.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#107

Post by xela79 »

JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

>Crawling the web is not a competitive advantage for any of these AI companies,

?? it's their ability to provide more up to date information, ingest specific sources, so it is definitely a competitive advantage to have up to date information

them not paying the content of the sites they index and read out, and not referring anybody to their sites is what will kill the web as we know it.

for a website owner there is zero value of having their content indexed by AI bots. Zilch.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#108

Post by stego-tech »

danaris wrote: post #82
asimpletune wrote: post #81

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot. I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the mod…

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in. "Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to t…

Literally this. It’s why I advocate for regulations over technological solutions nowadays.

We have all the technology we need to solve today’s ills (or support the R&D needed to solve today’s ills). The problem is that this technology isn’t being used to make life better, just more extractive of resources from those without towards those who have too much. The solution to that isn’t more technology (France already PoC’ed the Guillotine, after all), but more regulations that eliminate loopholes and punish bad actors while preserving the interests of the general public/commons.

Bad actors can’t be innovated away with new technological innovations; the only response to them has always been rules and punishments.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#109

Post by skybrian »

JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

Although it doesn’t actually build the index, if AI crawlers really do want to save on crawling costs, couldn’t they share a common index? Seems like it’s up to them to build it.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#110

Post by 1dom »

imglorp wrote: post #104
1dom wrote: post #98

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl. On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better. This…

Yes, right, it should be an open protocol so any CDN or content provider can use it the same way. Hopefully it becomes a part of popular web servers so little guys can play along without a CDN. It needn't be crypto, but would be convenient. Lacking that, it would need some unforgeable presentation of identity that could be connected to a bank account. I shed not one tear for the crawlers - they had their chance to re…

I can't work out how an open protocol implementation of this could work without crypto: ultimately if it's just fiat, a business entity needs to be the payment processor who aggregates microtransactions and pays them out to content owners, this is the role cloudflare is playing.

The problem is microtransactions are not feasible in fiat, and to remove the aggregator role like Cloudflare means a huge amount of microtransactions from each potential crawler to each content owner. That's just too much expensive work compared to the current position.

I agree though, I shed no tears for crawlers, but hopefully we're beyond the naivety of honour systems - again, the thing crypto was supposed to be solving.

Forcing big evil crawler entities to bend the knee by hiding behind big evil CDN entities feels silly though.

Post reply on HN