Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

681–690 of 712 posts

Re: Creepy Crawlies

#681

Post by alkonaut »

Arubis wrote: post #656
alkonaut wrote: post #298

Earlier quoted context omitted.

Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving. I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perha…

> Every place in earth has a legal definition of who is human. I actually doubt that! And in places that do I doubt it’s perfection. Citizenship is mostly defined legally. Residency has a legal definition that may or may not track with reality. Most legal regimes I’m aware of (IANAL) define personhood; whether humanness is defined I’m less sure of. Things that are more nebulous and subjective…sometimes they’re define…

Issuing this digital ID for all the _other things_ I do with it already has to deal with all this!

Of course the debate would revolve around ”is it fine to have a national government issued ID be the ultimate proof of your identity/humanity?”

It’s problematic - yet I think it’s better than the alternative.

Re: Creepy Crawlies

#682

Post by GoblinSlayer »

akoboldfrying wrote: post #541
alok-g wrote: post #531

Earlier quoted context omitted.

I had to look up what iocaine route is. Here's an article that explains well: https://firesphere.dev/articles/iocaine-the-deadliest-poison...

That was somewhat helpful, thanks, but the most interesting aspect for me is how iocaine tries to determine whether a request comes from a bot (since if this can be done reliably, we're done) -- but on that, all the link says is that it "uses a set of logical rules".

That link shows me:

---

Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a.

Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He.

---

Re: Creepy Crawlies

#683

Post by shakna »

inigyou wrote: post #668
shakna wrote: post #599

Earlier quoted context omitted.

... It also has a quote from Anthropic, admitting they backed off their scraper after a robots.txt update.

Nobody who gives a shit about their software actually working obeys robots.txt. most robots.txt block everything except googlebot, which is an insane policy and you don't have to follow it.

Which isn't all that relevant a discussion here. Anthropic saying they did so in this case, means that they admit they were the scraper flooding the domain.

Re: Creepy Crawlies

#684

Post by fc417fc802 »

Dylan16807 wrote: post #679
fc417fc802 wrote: post #536

Earlier quoted context omitted.

> does it actually raise the cost or reduce asymmetry in a meaningful way? It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough. PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat ou…

But a couple orders of magnitude is still a ton of asymmetry. And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.

Sure. Even if PoW in general is sound, when applied to scraping it's always going to be a matter of what the client is willing to "pay". It's entirely within the realm of possibility that a scraper could be willing to "outbid" the user of a 5 year old tablet.

But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.

To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.

I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?

Re: Creepy Crawlies

#685

Post by DarmokTanagra »

Wowfunhappy wrote: post #433

Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?

three guesses what geographic regions they are based in.

Re: Creepy Crawlies

#687

Post by Dylan16807 »

fc417fc802 wrote: post #684
Dylan16807 wrote: post #679

Earlier quoted context omitted.

But a couple orders of magnitude is still a ton of asymmetry. And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.

Sure. Even if PoW in general is sound, when applied to scraping it's always going to be a matter of what the client is willing to "pay". It's entirely within the realm of possibility that a scraper could be willing to "outbid" the user of a 5 year old tablet. But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to m…

> why are they running such lightweight setups to begin with?

It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.

> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.

I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.

And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.

Re: Creepy Crawlies

#688

Post by GoblinSlayer »

xena wrote: post #325
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...

Was a post form tried as a challenge?

[text field][submit]

(type qwerty and submit)

Re: Creepy Crawlies

#689

Post by GoblinSlayer »

LatticeAnimal wrote: post #152
semiquaver wrote: post #126

Earlier quoted context omitted.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

hashx kind of already does it.

Re: Creepy Crawlies

#690

Post by sgsjchs »

dewey wrote: post #624
sgsjchs wrote: post #410

Ironically, defense by obscurity may be the way to go here. Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it. You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or bu…

Render page in headless browser with JS, cache challenge solution, re-use on all your non-browser requests. Won't really change anything and it will auto-adapt to any new challenge that is being thrown.

It's not reusable from a different IP.
Post reply on HN