Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who does Anubis actually stop?

fzakaria.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: Who does Anubis actually stop?

#41

Post by 710dev »

what wrote: post #34
novafunc wrote: post #8

Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas. You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers. It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to no…

>tries not to be annoying The little anime girl is pretty off putting. I bounce when I see it.

[dead]

Re: Who does Anubis actually stop?

#43

Post by VladVladikoff »

TomatoCo wrote: post #7
jdlshore wrote: post #4

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis se…

There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.

In my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s really big!)

Re: Who does Anubis actually stop?

#44

Post by Macha »

greyface- wrote: post #33
Macha wrote: post #25

Earlier quoted context omitted.

And then people will move to a new solution. I think this is a pragmatist vs idealist debate. The pragmatic answer is that Anubis solves a problem now, and so it will be used until either it doesn't or a better solution presents itself. The idealist approach is that it's obvious that there's ways to get around Anubis, and so some people argue from there that it shouldn't be used. But the only other alternatives being…

Anubis is the penicillin of web hosting. Every individual practitioner has a strong incentive to overuse it, because it's extremely effective for them individually. Every additional practitioner that uses it increases the selection pressure on their collective adversary to develop resistance. Eventually, it reaches a tipping point and becomes ineffective. But the ecosystem-level impact of its historical use remains,…

Viruses do not have motivations, so it’s hard to assign blame to them. The companies running badly behaving AI scrapers are run by people with more of a mind than viruses, so please direct your complaints about the second order effects of their actions that way, rather than on their direct victims.

Re: Who does Anubis actually stop?

#45

Post by Rendello »

aboardRat4 wrote: post #18
throw0101d wrote: post #13

I was curious about the name: > Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots. * https://anubis.techaro.lol/docs/ > The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice respo…

>>I was curious about the name: That's knowledge usually learnt in primary school.

https://xkcd.com/1053/

Re: Who does Anubis actually stop?

#46

Post by inigyou »

jdlshore wrote: post #4

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis se…

It does it by blocking the scrapers because they are really stupid scrapers. go-away blocks them too, by seeing if they load images, a much quicker test.

Re: Who does Anubis actually stop?

#47

Post by inigyou »

VladVladikoff wrote: post #43
TomatoCo wrote: post #7

Earlier quoted context omitted.

There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.

In my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s…

And each new IP triggers a new Anubis challenge. Isn't it great?

Re: Who does Anubis actually stop?

#48

Post by inigyou »

xboxnolifes wrote: post #22
jsnell wrote: post #12

Earlier quoted context omitted.

Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests. How big of a deterrent is a millionth of a cent per page goi…

Your theoretical counterargument falls apart by the reality of just putting up anubis and comparing the before and after. I don't understand why this argument shows up in every thread about anubis. There are plenty of people and orgs who have empirical before and after results. We don't need theoretical arguments when there exists actual data.

It's not because of the PoW though, it's just because the scraper doesn't run JavaScript. The alternative package called go-away does these tests without the PoW.

Re: Who does Anubis actually stop?

#49

Post by ssl-3 »

nitwit005 wrote: post #14

This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping. The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.

There are multiple goals at play. They're easy to find in HN comment sections whenever these topics arise.

One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all.

Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web.

These are not the same goals.

Re: Who does Anubis actually stop?

#50

Post by inigyou »

greyface- wrote: post #20
Macha wrote: post #15

Earlier quoted context omitted.

Practically, the people indiscriminately scraping don't bother to do the work to bypass it or implement the POW test, which results in reduced CPU load for all the properties that were having trouble with scrapers before. Until scrapers start implementing it en masse then, it still serves its purpose.

> Until scrapers start implementing it en masse If it becomes widespread (as it has been doing), they will. Anubis' strategy only works while it remains a niche approach only adopted by a small number of sites.

Then they'll change what Anubis does.
Post reply on HN