Site doesn't load, must be hit by AI crawlers.
Why are anime catgirls blocking my access to the Linux kernel?
451–460 of 968 posts
Re: Why are anime catgirls blocking my access to the Linux kernel?
#452I have a S24 (flagship of 2024) and Anubis often takes 10-20 seconds to complete, that time is going to add up if more and more sites adopt it, leaning to a worse browsing experience and wasted battery life. Meanwhile AI farms will just run their own nuclear reactors eventually and be unaffected. I really don't understand why someone thought this was a good idea, even if well intentioned.
Something is wrong with your flagship if it takes that long.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#453i suppose one nice property is that it is trivially scalable. if the problem gets really bad and the scrapers have llms embedded in them to solve captchas, the difficulty could be cranked up and the lifetime could be cranked down. it would make the user experience pretty crappy (party like it's 1999) but it could keep sites up for unauthenticated users without engaging in some captcha complexity race.
it does have arty political vibes though, the distributed and decentralized open source internet with guardian catgirls vs. late stage capitalism's quixotic quest to eat itself to death trying to build an intellectual and economic robot black hole.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#454Earlier quoted context omitted.
OpenAI/Anthropic/Perplexity aren't the bad actors here. If they are, they are relatively simply to block - why would you implement an Anubis PoW MITM Proxy, when you could just simply block on UA? I get the sense many of the bad actors are simply poor copycats that are poorly building LLMs and are scraping the entire web without a care in the world
> why would you implement an Anubis PoW MITM Proxy, when you could just simply block on UA? That's in fact what I was asking: I've only seen traffic from these kind of companies and I've easily blocked them without an annoying PoW scheme. I have yet to see any of these bad actors and I'm interested in knowing who they actually are.
Huawei. Be happy that you haven't been hit by them yet.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#455Earlier quoted context omitted.
I don't think there are any. Because servers would serve different content based on user agent virtually all browsers start with Mozilla/5.0...
curl, wget, lynx, and elinks all don't by default (I checked). Mainstream web browsers likely all do, and will forever.
Anubis will let curl through, while blocking any non-mainstream browser which will likely say "Mozilla" in its UA just for best compatibility and call that a "bot"? WTF.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#456Earlier quoted context omitted.
I'm saying your notion of "the same cost" is off. They pay the same total CPU cost, but that isn't the actual perceived cost in each case.
Can you flesh that out more? In the case of AI scrapers it seems especially clear: the model companies just want tokens, and are paying a (one-time) cost of C for N tokens. Again, with Hashcash, this isn't how it works: most outbound spam messages are worthless. The point of the system is to exploit the negative exponent on the attacker's value function.
The scraper breaking every time a new version of Anubis is deployed, until new anti-Anubis features are implemented, is the point; if the scrapers were well-engineered by a team that cared about the individual sites they're scraping, they probably wouldn't be so pathological towards forges.
The human-labor cost of working around Anubis is unlikely to be paid unless it affects enough data to be worth dedicating time to, and the data they're trying to scrape can typically be obtained "respectfully" in those cases -- instead of hitting the git blame route on every file of every commit of every repo, just clone the repos and run it locally, etc.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#457TFA — and most comments here — seem to completely miss what I thought was the main point of Anubis: it counters the crawler's "identity scattering"/sybil'ing/parallel crawling. Any access will fall into either of the following categories: - client with JS and cookies. In this case the server now has an identity to apply rate limiting to, from the cookie. Humans should never hit it, but crawlers will be slowed down im…
You don't necessarily need JS, you just need something that can detect if Anybis is used and complete the challenge.
[deleted]
Re: Why are anime catgirls blocking my access to the Linux kernel?
#458Earlier quoted context omitted.
It's more likely that the project itself will disappear into irrelevance as soon as AI scrapers bother implementing the PoW (which is trivial for them, as the post explains) or figure out that they can simply remove "Mozilla" from their user-agent to bypass it entirely.
[flagged]
I thought HN was anti-copyright and anti-imaginary-property, or at least the bulk of its users were. Yet all of a sudden, "but AI!!!!1"?
a federal crime
The rest of the world doesn't care.
Re: Why are anime catgirls blocking my access to the Linux kernel?
#459Earlier quoted context omitted.
I think the argument on offer is more, this juice isn't worth the squeeze. Each user is being slowed down and annoyed for something that bots will trivially bypass if they become aware of it.
If they become aware of it and actually think it’s worthwhile. Malicious bots work by scaling, and implementing special cases for every random web site doesn’t scale. And it’s likely they never even notice.
If this kind of security by not being noticed is the plan, why not just have a trivial (but unique) captcha that asks the user to click a button with no battery wasting computation?
Re: Why are anime catgirls blocking my access to the Linux kernel?
#460Earlier quoted context omitted.
This arms race will have a terminus. The bots will eventually be indistinguishable from humans. Some already are.
> The bots will eventually be indistinguishable from humans Not until they get issued government IDs they won't! Extrapolating from current trends, some form of online ID attestation (likely based on government-issued ID[1]) will become normal in the next decade, and naturally, this will be included in the anti-bot arsenal. It will be up to the site operator to trust identities signed by the Russian government. 1. De…
This has quite nasty consequences for privacy. For this reason, alternatives are desirable. I have less confidence on what such an alternative should be, however.