Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

671–680 of 711 posts

Re: Creepy Crawlies

#671

Post by teravor »

crote wrote: post #568
teravor wrote: post #483

Earlier quoted context omitted.

anubis was a joke on arrival. PoW for this purpose is plain stupid. the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.

Yes, but it actually works . As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic! It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it…

that means all you had to do was gate the content behind a javascript snippet with a timer.

Re: Creepy Crawlies

#672

Post by roughly »

inigyou wrote: post #665
roughly wrote: post #652

Earlier quoted context omitted.

Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.

How will you know anything about a system you stubbornly only look at one side of? Like saying planes are terrible because they're loud. They are, but have you never traveled to a different city?

I’ve never mugged someone either, I suppose I should give it a shot before I go passing judgement.

Re: Creepy Crawlies

#673

Post by articulatepang »

inigyou wrote: post #230
paytonjjones wrote: post #203

Earlier quoted context omitted.

It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and sc…

Why aren't we litigating against scrapers, anyway? DDoS is a felony.

Who’s “we”? I don’t want to do that work. I don’t think kernel maintainers do, either. Do you?

Re: Creepy Crawlies

#674

Post by darkmighty »

GoblinSlayer wrote: post #670
darkmighty wrote: post #635

Earlier quoted context omitted.

Maybe, but at least you can more easily assign responsibility in this case. If you soft-block or hard-block someone because they're in a botnet, you can get them to change (expire the old ones) their credentials and start with clean systems. Maybe they'll realize their TV is part of a botnet if they have to keep doing it, or their PC has malware.

The pattern described in the blog is that one ip makes 4-5 requests and never comes back.

Indeed, but then you can (for example) just basically blocklist this ID and share this blocklist with other servers (and likewise import blocklists) -- a blocklist curated by one or more NGOs or the sort would be ideal.

Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done).

Also, bots cannot simply automate renewing the credentials of course, after expiry you should be required to physically (or in a trustworthy digital medium) re-authenticate yourself preventing any kind of mass attack, you'd need millions of people to perform authentication multiple times a day (by that point it seems safe to assume they'd rather fix the malware source in their personal devices).

Re: Creepy Crawlies

#675

Post by andai »

Demiurge wrote: post #636
andai wrote: post #627

Earlier quoted context omitted.

> Are the AI companies worried about anything but their valuations? >I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large. Doesn't the same thing apply to every company? And also every government? And also individual? We're all doing the "locally rational" thing, and globally doing... well... I think the main fa…

Yes, this applies to every company. The institutions have evolved a lot, and how we look at social safety and fairness have both evolved and devolved at the same time. Companies, individuals, and governments have different roles. Rationality from their perspective is specific to their perspective, and I don’t believe just because evolution or physics are rational we should die off from a new virus or asteroid impact.…

>physics is rational

What is its objective function?

Re: Creepy Crawlies

#676

Post by xena »

darkmighty wrote: post #674
GoblinSlayer wrote: post #670

Earlier quoted context omitted.

The pattern described in the blog is that one ip makes 4-5 requests and never comes back.

Indeed, but then you can (for example) just basically blocklist this ID and share this blocklist with other servers (and likewise import blocklists) -- a blocklist curated by one or more NGOs or the sort would be ideal. Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done). Also, bots cannot simply automate renewing th…

I am actually working on this kind of shared reputation database: https://github.com/TecharoHQ/reputationdb (warning: very prototype, not documented, such bugs, wow). I just have more to do than I have free time at this point.

Re: Creepy Crawlies

#677

Post by GoblinSlayer »

cyberax wrote: post #225
tptacek wrote: post #118

Earlier quoted context omitted.

This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scraper…

I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute. Anubis doesn't really help against this.

This, blocking granularity is a country, not ip address.

Re: Creepy Crawlies

#678

Post by edward28 »

skybrian wrote: post #124
chuckadams wrote: post #8

Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.

Apparently the current implementation is very inefficient by modern standards: https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.

Maybe it's time for AI to rewrite it in rust.

Re: Creepy Crawlies

#679

Post by Dylan16807 »

fc417fc802 wrote: post #536
Dylan16807 wrote: post #504

Earlier quoted context omitted.

That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance. I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?

> does it actually raise the cost or reduce asymmetry in a meaningful way? It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough. PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat ou…

But a couple orders of magnitude is still a ton of asymmetry.

And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.

Re: Creepy Crawlies

#680

Post by Demiurge »

andai wrote: post #675
Demiurge wrote: post #636

Earlier quoted context omitted.

Yes, this applies to every company. The institutions have evolved a lot, and how we look at social safety and fairness have both evolved and devolved at the same time. Companies, individuals, and governments have different roles. Rationality from their perspective is specific to their perspective, and I don’t believe just because evolution or physics are rational we should die off from a new virus or asteroid impact.…

>physics is rational What is its objective function?

Entropy.
Post reply on HN