All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.
[deleted]
Cloudflare to introduce pay-per-crawl for AI bots
61–70 of 317 posts
Re: Cloudflare to introduce pay-per-crawl for AI bots
#62Earlier quoted context omitted.
Yes, but I do feel this makes "theft" arguments stronger if they're deliberately evading the paywall, if you decided to be litigious about it.
This isn't the kind of problem that really ought to be solved through courts. It's obvious to anyone that this is a new kind of problem that no author of the current jurisprudence envisioned. We need new legislation to stop this kind of abuse of the commons.
I strongly agree with you, but I have no confidence in my country's current elected representatives to ever do anything good, so our hands are tied until we vote them out.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#63This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402
This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…
crypto seems like a massive waste for what can just be a regular transaction
Re: Cloudflare to introduce pay-per-crawl for AI bots
#64What about if somebody uses artificial intelligence crawler to help them navigate the web as an accessibility tool?
Enabling UI automation. It already throws up a lot of... uh... troublesome verifications.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#65Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.
Google had an experiment called Google Contributor where you could buy all your own ads. This effectively had the experience you're describing (prepay and get no ads until it runs out). They tried it twice, so someone wanted it to work. I was always curious why they shuttered it.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#66Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.
>Google with get it for free because Google Search
What if the second step is that Google pays the page it visits? By enabling a crawler fee per page, news websites could make some articles uncrawlable unless a huge fee is paid. Just thinking aloud, but I could easily see a protocol stating pricing by different kinds of "licensing" e.g. "internal usage", "redistribution" (what google news did/does?), "LLM training", etc. Cloudflare, acting as a central point for millions of websites, makes this possible.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#67Earlier quoted context omitted.
This should be the standard business model on the web, instead of the advertising middlemen that have corrupted all our media, and the adtech that exploits our data in perpetuity. All of which is also serving to spread propaganda, corrupt democratic processes, and cause the sociopolitical unrest we've seen in the last decade+. I hope that decades from now we can accept how insidious all of this is, and prosecute and…
crypto seems like a massive waste for what can just be a regular transaction
Something like BAT isn't that wasteful, and without crypto you'd be stuck never getting paid by bad actors in the scheme.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#68Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.
If you don't want payment, there is:
Used by https://gcc.gnu.org/bugzilla/ for example. It is less annoying than CAPTCHA/Turnstile/whatever because the proof of work runs automatically.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#69Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.
Google had an experiment called Google Contributor where you could buy all your own ads. This effectively had the experience you're describing (prepay and get no ads until it runs out). They tried it twice, so someone wanted it to work. I was always curious why they shuttered it.
Tiny selection of web sites, restricted to the US, absolutely no marketing.
Re: Cloudflare to introduce pay-per-crawl for AI bots
#70All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.
This could get AI scrapers hit with a DMCA circumvention lawsuit, which is $2,500 / scrape + attorney fees of both sides if they lose.