Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are anime catgirls blocking my access to the Linux kernel?

lock.cmpxchg8b.com

Post reply on HN
Print view View original

41–50 of 968 posts

Re: Why are anime catgirls blocking my access to the Linux kernel?

#41

Post by majorchord »

WesolyKubeczek wrote: post #19
rnhmjoj wrote: post #16

I don't understand, why do people resort to this tool instead of simply blocking by UA string or IP address. Are there so many people running these AI crawlers? I blackholed some IP blocks of OpenAI, Mistral and another handful of companies and 100% of this crap traffic to my webserver disappeared.

You should read more. AI companies use residential proxies and mask their user agents with legitimate browser ones, so good luck blocking that.

> AI companies use residential proxies

Source:

Re: Why are anime catgirls blocking my access to the Linux kernel?

#42

Post by slowmovintarget »

ranger_danger wrote: post #40
immibis wrote: post #35

Earlier quoted context omitted.

Why haven't they been sued and jailed for DDoS, which is a felony?

Criminal convictions in the US require a standard of proof that is "beyond a reasonable doubt" and I suspect cases like this would not pass the required mens rea test, as, in their minds at least (and probably a judge's), there was no ill intent to cause a denial of service... and trying to argue otherwise based on any technical reasoning (e.g. "most servers cannot handle this load and they somehow knew it") is IMO u…

I thought only capital crimes (murder, for example) held the standard of beyond a reasonable doubt. Lesser crimes require the standard of either a "Preponderance of Evidence" or "Clear and Convincing Evidence" as burden of proof.

Still, even by those lesser standards, it's hard to build a case.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#43

Post by fluoridation »

jsnell wrote: post #39
fluoridation wrote: post #36

Earlier quoted context omitted.

>An hour of a server CPU costs $0.01. How much is an hour of your time worth? That's irrelevant. A human is not going to be solving the challenge by hand, nor is the computer of a legitimate user going to be solving the challenge continuously for one hour. The real question is, does the challenge slow down clients enough that the server does not expend outsized resources serving requests of only a few users? >Even if…

The human needs to wait for their computer to solve the challenge. You are trading something dirt-cheap (CPU time) for something incredibly expensive (human latency). Case in point: > If the challenge takes, say, 250 ms on the absolute best hardware, and serving a request takes 25 ms, a normal user won't even see a difference, while a scraper will see a tenfold slowdown while scraping that website. No. A human sees a…

>No. A human sees a 10x slowdown.

For the actual request, yes. For the complete experience of using the website not so much, since a human will take at least several seconds to process the information returned.

>And the scraper paid one 1/1000000th of a dollar. (The scraper does not care about latency.)

The point need not be to punish the client, but to throttle it. The scraper may not care about taking longer, but the website's operator may very well care about not being hammered by requests.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#44

Post by kingstnap »

iefbr14 wrote: post #26

I wouldn't be surprised if just delaying the server response by some 3 seconds will have the same effect on those scrapers as Anubis claims.

There is literally no point wasting 3 seconds of a computer's time and it's expensive wasting 3 seconds of a person's time.

That is literally an anti-human filter.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#45

Post by Arnavion »

>This dance to get access is just a minor annoyance for me, but I question how it proves I’m not a bot. These steps can be trivially and cheaply automated.

>I think the end result is just an internet resource I need is a little harder to access, and we have to waste a small amount of energy.

No need to mimic the actual challenge process. Just change your user agent to not have "Mozilla" in it; Anubis only serves you the challenge if it has that. For myself I just made a sideloaded browser extension to override the UA header for the handful of websites I visit that use Anubis, including those two kernel.org domains.

(Why do I do it? For most of them I don't enable JS or cookies for so the challenge wouldn't pass anyway. For the ones that I do enable JS or cookies for, various self-hosted gitlab instances, I don't consent to my electricity being used for this any more than if it was mining Monero or something.)

Re: Why are anime catgirls blocking my access to the Linux kernel?

#47

Post by eurleif »

slowmovintarget wrote: post #42
ranger_danger wrote: post #40

Earlier quoted context omitted.

Criminal convictions in the US require a standard of proof that is "beyond a reasonable doubt" and I suspect cases like this would not pass the required mens rea test, as, in their minds at least (and probably a judge's), there was no ill intent to cause a denial of service... and trying to argue otherwise based on any technical reasoning (e.g. "most servers cannot handle this load and they somehow knew it") is IMO u…

I thought only capital crimes (murder, for example) held the standard of beyond a reasonable doubt. Lesser crimes require the standard of either a "Preponderance of Evidence" or "Clear and Convincing Evidence" as burden of proof. Still, even by those lesser standards, it's hard to build a case.

No, all criminal convictions require proof beyond a reasonable doubt: https://constitution.congress.gov/browse/essay/amdt14-S1-5-5...

>Absent a guilty plea, the Due Process Clause requires proof beyond a reasonable doubt before a person may be convicted of a crime.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#48

Post by Macha »

majorchord wrote: post #41
WesolyKubeczek wrote: post #19

Earlier quoted context omitted.

You should read more. AI companies use residential proxies and mask their user agents with legitimate browser ones, so good luck blocking that.

> AI companies use residential proxies Source:

Source: Cloudflare

https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Perplexity's defense is that they're not doing it for training/KB building crawls but for answering dynamic queries calls and this is apparently better.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#49

Post by dilDDoS »

jimmaswell wrote: post #12

What exactly is so bad about AI crawlers compared to Google or Bing? Is there more volume or is it just "I don't like AI"?

As others have said, it's definitely volume, but also the lack of respecting robots.txt. Most AI crawlers that I've seen bombarding our sites just relentlessly scrape anything and everything, without even checking to see if anything has changed since the last time they crawled the site.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#50

Post by johnea »

My biggest bitch is that it requires JS and cookies...

Although the long term problem is the business model of servers paying for all network bandwidth.

Actual human users have consumed a minority of total net bandwidth for decades:

https://www.atom.com/blog/internet-statistics/

Part 4 shows bots out using humans in 1996 8-/

What are "bots"? This needs to include goggleadservices, PIA sharing for profit, real-time ad auctions, and other "non-user" traffic.

The difference between that and the LLM training data scraping, is that the previous non-human traffic was assumed, by site servers, to increase their human traffic, through search engine ranking, and thus their revenue. However the current training data scraping is likely to have the opposite effect: capturing traffic with LLM summaries, instead of redirecting it to original source sites.

This is the first major disruption to the internet's model of finance since ad revenue look over after the dot bomb.

So far, it's in the same category as the environmental disaster in progress, ownership is refusing to acknowledge the problem, and insisting on business as usual.

Rational predictions are that it's not going to end well...

Post reply on HN