Content Independence Day: no AI crawl without compensation
blog.cloudflare.com
Content Independence Day: no AI crawl without compensation
1–10 of 39 posts
Re: Content Independence Day: no AI crawl without compensation
#2I don’t understand, companies that want to crawl can pay for services like brightdata or crawlbase, the barriers don’t apply to them
This ends up hurting individuals and small companies that are harmless and cannot afford to pay
Re: Content Independence Day: no AI crawl without compensation
#3It's unfortunate but I think the ship has sailed. Good on them for trying but I don't see it working.
I am advising all my clients away from informational content which is easily remixed by LLMs. And I'm not bothering anymore with targeting informational search queries on my own sites.
I'm doubling down on community and interaction. Finding ways to interact with original content with smaller audiences, rather than produce information for a global search audience.
Re: Content Independence Day: no AI crawl without compensation
#4A nice attempt and another layer for the swiss cheese of technology it will take to try and ease the burden AI companies are putting on people trying to run websites.
I'd be cautious about relying on just the good will of Cloudflare.
It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Internet.
Re: Content Independence Day: no AI crawl without compensation
#5> Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers
How is this done, technically? User agent checking? IP range blocking?
Re: Content Independence Day: no AI crawl without compensation
#6> Imagine an AI engine like a block of swiss cheese. New, original content that fills one of the holes in the AI engine’s block of cheese is more valuable than repetitive, low-value content that unfortunately dominates much of the web today.
Great statement in theory - but in practice, the whole people-as-a-service industry for AI data generation is IMO more damaging to the knowledge ecosystem than open data. e.g. companies like pareto.ai
"Proprietary data for pennies on the dollar" is the late-stage capitalism equivalent of the postdoctoral research trap.
Re: Content Independence Day: no AI crawl without compensation
#7> Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers How is this done, technically? User agent checking? IP range blocking?
Details here: https://blog.cloudflare.com/introducing-pay-per-crawl/
Re: Content Independence Day: no AI crawl without compensation
#8A nice attempt and another layer for the swiss cheese of technology it will take to try and ease the burden AI companies are putting on people trying to run websites. I'd be cautious about relying on just the good will of Cloudflare. It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Int…
Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances.
Definitely don't rewrite your web server more efficiently in Rust instead. /s
Re: Content Independence Day: no AI crawl without compensation
#9Currently, what I do is that when an IP requests insane amounts of URLs on my server (especially when its all broken urls causing 404s) I look up the IP and then block the whole organization.
For example today some bot from the range 14.224.0.0-14.255.255.255 got crazy and caused a storm of 404s. Dozens per second for hours on end. So I blocked the range like this:
iptables -A INPUT -m iprange --src-range 14.224.0.0-14.255.255.255 -j DROP
That's probably not the best way and might block significant parts of whole countries. But at least it keeps my service alive for now.
What do others here do to protect their servers?
Re: Content Independence Day: no AI crawl without compensation
#10> Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers How is this done, technically? User agent checking? IP range blocking?
Details here: https://blog.cloudflare.com/introducing-pay-per-crawl/
This requires good faith on behalf of the crawler? So it's DOA; why even bother implementing this?
Also, what a piece of zero-trust shit the web is becoming thanks to a couple of shit heads who really need to extract monetary value out of everything. Even if this non-solution were to work, the prospect of putting every website behind Cloudsnare is not a good one anyway.
What the web needs right now, to be honest, is machetes. In ample quantity. Tell me who's running that crawler that is bothering you and I will put them to the sword. They won't even need to present a JWK in the header.