We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…
Proof-of-work works in terms of preventing large-scale automation. As for letting well behaved crawlers in, I've had an idea for something like DKIM for crawlers. Should be possible to set up a fairly cheap cryptographic solution that enables crawlers a persistent identity that can't be forged. Basically put a header containing first a string including today's date, the crawler's IP, and a domain name, then a cryptog…
It wouldn't work to prevent the type of behavior shown in a title story