Is there a cut that cloudfare gets or is that behind an NDA?
Content Independence Day: no AI crawl without compensation
11–20 of 39 posts
Re: Content Independence Day: no AI crawl without compensation
#12This will play out precisely like the "do not track" header; bad actors will create an arms race that makes anyone respecting it into a chump.
Re: Content Independence Day: no AI crawl without compensation
#13Earlier quoted context omitted.
Details here: https://blog.cloudflare.com/introducing-pay-per-crawl/
This requires good faith on behalf of the crawler? So it's DOA; why even bother implementing this? Also, what a piece of zero-trust shit the web is becoming thanks to a couple of shit heads who really need to extract monetary value out of everything. Even if this non-solution were to work, the prospect of putting every website behind Cloudsnare is not a good one anyway. What the web needs right now, to be honest, is…
Maybe I didn't understand the proposal completely yet, but wouldn't the crawler only have to cooperate (send the right headers, implement that auth framework, etc) if they want to pay?
The standard response to a crawler is a 402 Payment Required response, probably as a result of an aggressive bot detection.
So essentially, it's turning a site's entire content into an API: Either sign up for an API key or get blocked.
The question remains though how well they will be able to distinguish bot traffic from humans - also, will they make an exception for search engines?
Re: Content Independence Day: no AI crawl without compensation
#14I thought the web was supposed to be free and open?
Re: Content Independence Day: no AI crawl without compensation
#15Discussion:
Cloudflare to introduce pay-per-crawl for AI bots
Re: Content Independence Day: no AI crawl without compensation
#16So are they going to try and IP gate them or trust that AI companies that literally stole the info they used to make the base models will now respect robots.txt entries?
Re: Content Independence Day: no AI crawl without compensation
#17A nice attempt and another layer for the swiss cheese of technology it will take to try and ease the burden AI companies are putting on people trying to run websites. I'd be cautious about relying on just the good will of Cloudflare. It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Int…
Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances. Definitely don't rewrite your web server more efficiently in Rust instead. /s
Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.
Re: Content Independence Day: no AI crawl without compensation
#18Would this be preferable to something like Anubis?
Re: Content Independence Day: no AI crawl without compensation
#19I thought the web was supposed to be free and open?
That ship has sailed
Re: Content Independence Day: no AI crawl without compensation
#20This will play out precisely like the "do not track" header; bad actors will create an arms race that makes anyone respecting it into a chump.
If the only way to escape a proof of work tarpit is to pay the toll, you’re either going to pay in money or time & compute.