Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who does Anubis actually stop?

fzakaria.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Who does Anubis actually stop?

#31

Post by PunchyHamster »

Macha wrote: post #15
drum55 wrote: post #11

Earlier quoted context omitted.

Claude Opus can make an optimized version of the proof of work solver that’s more than 10000x faster than the javascript one, in about 5 minutes time. Who is this stopping exactly? It’s the wrong tool in the wrong place.

Practically, the people indiscriminately scraping don't bother to do the work to bypass it or implement the POW test, which results in reduced CPU load for all the properties that were having trouble with scrapers before. Until scrapers start implementing it en masse then, it still serves its purpose.

That was true but with AI that could be automated pretty easily. Sure, not worth for random blog but random blog won't get the traffic anyway

Re: Who does Anubis actually stop?

#32

Post by dullcrisp »

TomatoCo wrote: post #7
jdlshore wrote: post #4

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis se…

There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.

If compute isn’t the bottleneck then it wouldn’t make it 100x more expensive to scrape. Of course if it does, then it’ll be effective.

Re: Who does Anubis actually stop?

#33

Post by greyface- »

Macha wrote: post #25
greyface- wrote: post #20

Earlier quoted context omitted.

> Until scrapers start implementing it en masse If it becomes widespread (as it has been doing), they will. Anubis' strategy only works while it remains a niche approach only adopted by a small number of sites.

And then people will move to a new solution. I think this is a pragmatist vs idealist debate. The pragmatic answer is that Anubis solves a problem now, and so it will be used until either it doesn't or a better solution presents itself. The idealist approach is that it's obvious that there's ways to get around Anubis, and so some people argue from there that it shouldn't be used. But the only other alternatives being…

Anubis is the penicillin of web hosting.

Every individual practitioner has a strong incentive to overuse it, because it's extremely effective for them individually. Every additional practitioner that uses it increases the selection pressure on their collective adversary to develop resistance. Eventually, it reaches a tipping point and becomes ineffective. But the ecosystem-level impact of its historical use remains, and makes things worse for everyone.

Recently, the baseline expectations of the Web have shifted, and I need to enable JavaScript to read a non-CloudFlare-proxied simple HTML site like LKML. I do not expect this shift will revert once Anubis outlives its effectiveness.

Re: Who does Anubis actually stop?

#34

Post by what »

novafunc wrote: post #8

Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas. You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers. It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to no…

>tries not to be annoying

The little anime girl is pretty off putting. I bounce when I see it.

Re: Who does Anubis actually stop?

#35

Post by Kuinox »

drum55 wrote: post #27
Kuinox wrote: post #16

Earlier quoted context omitted.

The difficulty is increased when the server get heavy traffic.

Then the only people who can use it are the ones with a GPU implementation.

No, the value of scrapping the page is long gone and the scrapper would have moved on. The users would still access the page by waiting a bit.

Re: Who does Anubis actually stop?

#36

Post by recursivecaveat »

lrem wrote: post #3

Uh wait, a cookie that can be reused for a time to fetch the rest of the content? That's the exact opposite of what I'd intuitively want for this system...

If you behave yourself and reuse the same cookie/IP instead of trying to hide your identity, other tools can be used to block you if your request volume is crazy. It also means regular people only have to solve once.

Re: Who does Anubis actually stop?

#37

Post by arn3n »

what wrote: post #34
novafunc wrote: post #8

Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas. You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers. It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to no…

>tries not to be annoying The little anime girl is pretty off putting. I bounce when I see it.

It’s meant to be both funny AND highly unprofessional; Anubis makes money of licensing a version of the firewall where you can change the image.

It’s a good strategy; personal websites and blogs can display the anime girl without fear, and companies that care about their image end up paying. Win-win.

Re: Who does Anubis actually stop?

#38

Post by Terr_ »

aboardRat4 wrote: post #18
throw0101d wrote: post #13

I was curious about the name: > Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots. * https://anubis.techaro.lol/docs/ > The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice respo…

>>I was curious about the name: That's knowledge usually learnt in primary school.

Grades 1-5 is a little hyperbolic, IIRC my "World Religions" class was probably middle/junior-high school.

In any case, it's not the type of lifetime common-knowledge which warrants your scornful response.

Re: Who does Anubis actually stop?

#39

Post by pibaker »

satvikpendem wrote: post #2

Exactly, it's the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.

A world without cloudflare does not necessarily mean a world where anyone can access any website. It could very well be a world where anyone can point a DDOS at any website and make it inaccessible to everyone.

Re: Who does Anubis actually stop?

#40

Post by TylerE »

xboxnolifes wrote: post #22
jsnell wrote: post #12

Earlier quoted context omitted.

Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests. How big of a deterrent is a millionth of a cent per page goi…

Your theoretical counterargument falls apart by the reality of just putting up anubis and comparing the before and after. I don't understand why this argument shows up in every thread about anubis. There are plenty of people and orgs who have empirical before and after results. We don't need theoretical arguments when there exists actual data.

Captchas worked for a few years too. Then computers got better at solving them than humans, and we're still dealing with the fucking useless things decades later.
Post reply on HN