Who does Anubis actually stop?
fzakaria.com
Who does Anubis actually stop?
1–10 of 84 posts
Re: Who does Anubis actually stop?
#2Exactly, it's the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.
Re: Who does Anubis actually stop?
#3Uh wait, a cookie that can be reused for a time to fetch the rest of the content? That's the exact opposite of what I'd intuitively want for this system...
Re: Who does Anubis actually stop?
#4Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely.
Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors.
(That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)
Re: Who does Anubis actually stop?
#5Funny, I'm working on a simple tool that pulls the atom feed of the latest patches from lore with Python, and I just slip a User-Agent header into the requests.get() and it works great.
Re: Who does Anubis actually stop?
#6> The exact adversary Anubis targets defeats it trivially.
Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive.
Anubis was not built to stop individual users with llms.
Re: Who does Anubis actually stop?
#7Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis se…
There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.
Re: Who does Anubis actually stop?
#8Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas.
You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers.
It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to not be too annoying for humans, to not block real users, and not be privacy invasive.
Re: Who does Anubis actually stop?
#9Exactly, it's the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.
Open to whom?
There can't be a truly open web if the guys with all the resources in the world have the incentive to absolutely crush you by draining all of your resources.
They won't crush you out of malice, but by accident, like an ant.
Re: Who does Anubis actually stop?
#10Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis se…
There is no distinction between the two. If one can do it the other can just as easily. I know multiple people right now that can scrape any a site very easily.
I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.