Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

121–130 of 694 posts

Re: Creepy Crawlies

#121

Post by lxgr »

TZubiri wrote: post #106
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental. When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.

I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero".

Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).

Re: Creepy Crawlies

#122

Post by leoqa »

It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.

Re: Creepy Crawlies

#123

Post by grep_it »

znnajdla wrote: post #58

Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

Did you read the article?

“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”

Re: Creepy Crawlies

#124

Post by skybrian »

chuckadams wrote: post #8

Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.

Apparently the current implementation is very inefficient by modern standards:

https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j

Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.

Re: Creepy Crawlies

#126

Post by semiquaver »

NobodyNada wrote: post #110
semiquaver wrote: post #101

Earlier quoted context omitted.

no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.

The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.

Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.

a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.

Re: Creepy Crawlies

#127

Post by lxgr »

NooneAtAll3 wrote: post #57

from what I see there are 2 solutions: 1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally 2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficu…

> straight up go to every representative there is

In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.

Re: Creepy Crawlies

#128

Post by graemep »

tptacek wrote: post #118
smallerize wrote: post #92

Earlier quoted context omitted.

But the scraper is making way more requests and is paying for all that compute.

This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scraper…

I wonder whether what Anubis is doing might be blocking agents that do not run JS or that have a timeout (or humans who give up as I just did)

Re: Creepy Crawlies

#129

Post by mmooss »

singpolyma3 wrote: post #76

Why is no one filing lawsuits over this yet?

Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it.

For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.

Re: Creepy Crawlies

#130

Post by lxgr »

nneonneo wrote: post #59
tptacek wrote: post #44

Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

> 66% of scrapers are turned away directly.

Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.

The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.

[1] https://news.ycombinator.com/item?id=49500040

Post reply on HN