Earlier quoted context omitted.
This was obviously dumb when it launched: 1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked. 2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and ge…
> This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources…
if you want to save some $$$ you can spend like 30 minutes making a cracker like in the article. just make it multi threaded, add a queue and boom, your scraper nodes can go back to their cheap configuration. or since these are AI orgs we're talking about, write a gpu cracker and laugh as it solves challenges far faster than any user could.
custom solutions aren't worth it for individual sites, but with how widespread anubis is it's become worth it.