Use the very fund to feed generated contents to LLM crawler is the only right move
Cloudflare to introduce pay-per-crawl for AI bots
31–40 of 317 posts
Re: Cloudflare to introduce pay-per-crawl for AI bots
#32"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee. And all of this does not stop the incumbents who have already stolen everything.
Cloudflare has not used CAPTCHAs since 2023: https://blog.cloudflare.com/turnstile-ga/
Re: Cloudflare to introduce pay-per-crawl for AI bots
#33This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost.
Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastructure.
Instead of all the different companies hitting sites independently, there should be a single crawler they all contribute to. They set up their filters and everybody whose filters match a URL contributes proportionately. They set up their transformations (e.g. HTML to Markdown; text to embeddings), and everybody who shares a transformation contributes proportionately.
This, in turn, would reduce the load on websites massively. Instead of everybody hitting the sites, just one crawler would. And instead of hoping that all the different crawlers obey robots.txt correctly, this can be enforced at a technical and contractual level. The clients just don’t get the blocked content delivered to them – and if they want to get it anyway, the cost of that is to implement and maintain their own crawler instead of using the shared resources of everybody else – something that is a lot more unattractive than just proxying through residential IPs.
And if you want to add payments on, sure, I guess. But I don’t think that’s going to get many people paid at all. Who is going to set up automated payments for content that hasn’t been seen yet? You’ll just be paying for loads of junk pages generated automatically.
There’s a solution here that makes it easier and cheaper to crawl for the AI companies and search engines, while reducing load on the websites and making blocking more effective. But instead, Cloudflare just went “nah, just pay up”. It’s pretty unimaginative and not the least bit compelling.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#34In theory, why not, in practice welcome to the world where neutrality of internet explode... Soon they could decide if your requests come from a specific company IP or networks, because you look suspicious... In addition, bot fighting was never supposed to be about blocking automatic users but about blocking abusers, like spammers and co. So now it means that bad actor can have a free pass if they pay (with stolen cr…
> In theory, why not, in practice welcome to the world where neutrality of internet explode...
Anybody that has the sense^Wgall to clear their cookies regularly lives in this world as you get that CloudFlare gate keeping for just about every site you visit.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#35Earlier quoted context omitted.
>pay $25/page for your AI A residential proxy is way cheaper for scraping.
Sure, and suing the AI companies that use residential proxies to get out of paying is expensive, but so are my other hobbies. Hilarity could ensue.
Well good luck with your endeavor in that case
Re: Cloudflare to introduce pay-per-crawl for AI bots
#36This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…
But don't these new costs create a direct incentive to cooperate?
Re: Cloudflare to introduce pay-per-crawl for AI bots
#37"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee. And all of this does not stop the incumbents who have already stolen everything.
Cloudflare has not used CAPTCHAs since 2023: https://blog.cloudflare.com/turnstile-ga/
Having to click "Verify you are a human" every single time is still awful.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#38This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…
I think you're looking at the wrong side of the market for the incentive structures here.
Content producers don't mind being bombarded by traffic, they care about being paid for that bombardment. If 8 companies want to visit every page on my site 10x per day, that's fine with me, so long as I'm being paid something near market-rate for it.
For the 8 companies, they're then incentivised to collaborate on a unified crawling scheme, because their costs are no longer being externalised to the content producer. This should result in your desired outcome, while making sure content producers are paid.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#39"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee. And all of this does not stop the incumbents who have already stolen everything.
Cloudflare has not used CAPTCHAs since 2023: https://blog.cloudflare.com/turnstile-ga/
To save a visit: they use turnstile, a captcha replacement. The checkbox with verify you are a human. I would call that a captcha, but it is debatable if a non-puzzle check is.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#40This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…
This would be a decent application of crypto, like brave is for micro payments.