Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are anime catgirls blocking my access to the Linux kernel?

lock.cmpxchg8b.com

Post reply on HN
Print view View original

11–20 of 968 posts

Re: Why are anime catgirls blocking my access to the Linux kernel?

#11

Post by davidclark »

Philpax wrote: post #6

The argument isn't that it's difficult for them to circumvent - it's not - but that it adds enough friction to force them to rethink how they're scraping at scale and/or self-throttle. I personally don't care about the act of scraping itself, but the volume of scraping traffic has forced administrators' hands here. I suspect we'd be seeing far fewer deployments if the scrapers behaved themselves to begin with.

The OP author shows that the cost to scrape an Anubis site is essentially zero since it is a fairly simple PoW algorithm that the scraper can easily solve. It adds basically no compute time or cost for a crawler run out of a data center. How does that force rethinking?

Re: Why are anime catgirls blocking my access to the Linux kernel?

#14

Post by hooverd »

davidclark wrote: post #11
Philpax wrote: post #6

The argument isn't that it's difficult for them to circumvent - it's not - but that it adds enough friction to force them to rethink how they're scraping at scale and/or self-throttle. I personally don't care about the act of scraping itself, but the volume of scraping traffic has forced administrators' hands here. I suspect we'd be seeing far fewer deployments if the scrapers behaved themselves to begin with.

The OP author shows that the cost to scrape an Anubis site is essentially zero since it is a fairly simple PoW algorithm that the scraper can easily solve. It adds basically no compute time or cost for a crawler run out of a data center. How does that force rethinking?

The problem with crawlers if that they're functionally indistinguishable from your average malware botnet in behavior. If you saw a bunch of traffic from residential IPs using the same token that's a big tell.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#15

Post by ksymph »

This is neither here nor there but the character isn't a cat. It's in the name, Anubis, who is an Egyptian deity typically depicted as a jackal or generic canine, and the gatekeeper of the afterlife who weighs the souls of the dead (hence the tagline). So more of a dog-girl, or jackal-girl if you want to be technical.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#16

Post by rnhmjoj »

I don't understand, why do people resort to this tool instead of simply blocking by UA string or IP address. Are there so many people running these AI crawlers?

I blackholed some IP blocks of OpenAI, Mistral and another handful of companies and 100% of this crap traffic to my webserver disappeared.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#17

Post by WesolyKubeczek »

I disagree with the post author in their premise that things like Anubis are easy to bypass if you craft your bot well enough and throw the compute at it.

Thing is, the actual lived experience of webmasters tells that the bots that scrape the internets for LLMs are nothing like crafted software. They are more like your neighborhood shit-for-brain meth junkies competing with one another who makes more robberies in a day, no matter the profit.

Those bots are extremely stupid. They are worse than script kiddies’ exploit searching software. They keep banging the pages without regard to how often, if ever, they change. If they were 1/10th like many scraping companies’ software, they wouldn’t be a problem in the first place.

Since these bots are so dumb, anything that is going to slow them down or stop them in their tracks is a good thing. Short of drone strikes on data centers or accidents involving owners of those companies that provide networks of botware and residential proxies for LLM companies, it seems fairly effective, doesn’t it?

Re: Why are anime catgirls blocking my access to the Linux kernel?

#18

Post by hooverd »

rnhmjoj wrote: post #16

I don't understand, why do people resort to this tool instead of simply blocking by UA string or IP address. Are there so many people running these AI crawlers? I blackholed some IP blocks of OpenAI, Mistral and another handful of companies and 100% of this crap traffic to my webserver disappeared.

less savory crawlers use residential proxies and are indistinguishable from malware traffic

Re: Why are anime catgirls blocking my access to the Linux kernel?

#19

Post by WesolyKubeczek »

rnhmjoj wrote: post #16

I don't understand, why do people resort to this tool instead of simply blocking by UA string or IP address. Are there so many people running these AI crawlers? I blackholed some IP blocks of OpenAI, Mistral and another handful of companies and 100% of this crap traffic to my webserver disappeared.

You should read more. AI companies use residential proxies and mask their user agents with legitimate browser ones, so good luck blocking that.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#20

Post by mnmalst »

rnhmjoj wrote: post #16

I don't understand, why do people resort to this tool instead of simply blocking by UA string or IP address. Are there so many people running these AI crawlers? I blackholed some IP blocks of OpenAI, Mistral and another handful of companies and 100% of this crap traffic to my webserver disappeared.

Because that solution simply does not work for all. People tried and the crawlers started using proxies with residential IPs.
Post reply on HN