Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are anime catgirls blocking my access to the Linux kernel?

lock.cmpxchg8b.com

Post reply on HN
Print view View original

391–400 of 968 posts

Re: Why are anime catgirls blocking my access to the Linux kernel?

#391

Post by heavyset_go »

codedokode wrote: post #198
johnecheck wrote: post #136

Earlier quoted context omitted.

Sadly, touching the user-agent header more or less instantly makes you uniquely identifiable. Browser fingerprinting works best against people with unique headers. There's probably millions of people using an untouched safari on iPhone. Once you touch your user-agent header, you're likely the only person in the world with that fingerprint.

If your headers are new every time then it is very difficult to figure out who is who.

It's very easy to train a model to identify anomalies like that.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#392

Post by bananalychee »

pilif wrote: post #389
hedora wrote: post #380

Earlier quoted context omitted.

This was obviously dumb when it launched: 1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked. 2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and ge…

> This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources…

Does it actually? I don't think I've seen a case study with hard numbers.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#393

Post by elcritch »

loloquwowndueo wrote: post #376
semiquaver wrote: post #367

Earlier quoted context omitted.

Rot13 a challenge string. It could be any arbitrary function.

That wouldn’t have the fallback rate-limiting functionality. It’s too cheap.

It’s too cheap as a rate limiter as it is if you read TFA.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#394

Post by ronsor »

bogwog wrote: post #56

I wonder if the best solution is still just to create link mazes with garbage text like this: https://blog.cloudflare.com/ai-labyrinth/ It won't stop the crawlers immediately, but it might lead to an overhyped and underwhelming LLM release from a big name company, and force them to reassess their crawling strategy going forward?

That won't work, because garbage data is filtered after the full dataset is collected anyway. Every LLM trainer these days knows that curation is key.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#395

Post by tptacek »

agwa wrote: post #315
johnklos wrote: post #244

This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be dismissive of Anubis. Sure, the people who make the AI scraper bots are going to figure out how to actually do the work. The point is that they hadn't, and this worked for quite a while. As the botmakers circumvent, new methods of proof-of-notbot will be made…

It sounds like you're saying that it's not the proof-of-work that's stopping AI scrapers, but the fact that Anubis imposes an unusual flow to load the site. If that's true Anubis should just remove the proof-of-work part, so legitimate human visitors don't have to stare at a loading screen for several seconds while their device wastes electricity.

Exactly this.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#396

Post by ack_complete »

userbinator wrote: post #331

As I've been saying for a while now - if you want to filter for only humans, ask questions only a human can easily answer; counting the number of letters in a word seems to be a good way to filter out LLMs, for example. Yes, that can be relatively easily gotten around, just like Anubis, but with the benefit that it doesn't filter out humans and has absolutely minimal system requirements (a browser that can submit HTM…

For smaller forums, any customization to the new account process will work. When I ran a forum that was getting a frustratingly high amount of spammer signups, I modified the login flow to ask the user to add 1 to the 6-digit number in the stock CAPTCHA. Spam signups dropped like a rock.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#397

Post by jchw »

MikeDVB wrote: post #366
jchw wrote: post #226

> This… makes no sense to me. Almost by definition, an AI vendor will have a datacenter full of compute capacity. It feels like this solution has the problem backwards, effectively only limiting access to those without resources or trying to conserve them. A lot of these bots consume a shit load of resources specifically because they don't handle cookies, which causes some software (in my experience, notably phpBB) t…

We have been seeing our clients' sites being absolutely *hammered* by AI bots trying to blend in. Some of the bots use invalid user agents - they _look_ valid on the surface, but under the slightest scrutiny, it becomes obvious they're not real browsers. Personally I have no issues with AI bots, that properly identify themselves, from scraping content as if the site operator doesn't want it to happen they can easily…

> We have been seeing our clients' sites being absolutely hammered by AI bots trying to blend in. Some of the bots use invalid user agents - they _look_ valid on the surface, but under the slightest scrutiny, it becomes obvious they're not real browsers.

Yep. I noticed this too.

> That said they could even run headless versions of the browser engines...

Yes, exactly. To my knowledge that's what's going on with the latest wave that is passing Anubis.

That said, it looks like the solution to that particular wave is going to be to just block Huawei cloud IP ranges for now. I guess a lot of these requests are coming from that direction.

Personally though I think there are still a lot of directions Anubis can go in that might tilt this cat and mouse game a bit more. I have some optimism.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#398

Post by kldg »

pilif wrote: post #389
hedora wrote: post #380

Earlier quoted context omitted.

This was obviously dumb when it launched: 1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked. 2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and ge…

> This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources…

does it work well? I run chromium controlled by playwright for scraping and typically make Gemini implement the script for it because it's not worth my time otherwise. -but I'm not crawling the Internet generally (which I think there is very little financial incentive to do; it's a very expensive process even ignoring Anubis et al); it's always that I want something specific and am sufficiently annoyed by lack of API.

regarding authentication mentioned elsewhere, passing cookies is no big deal.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#399

Post by ronsor »

tambourine_man wrote: post #368
bawolff wrote: post #253

Earlier quoted context omitted.

It doesnt take much to host a static website. Its all the dynamic stuff/frameworks/db/etc that bogs everything down.

Still, 128MB is not enough to even run Debian let alone Apache/NGINX. I’m on my phone, but it doesn’t seem like the author is using Cloudflare or another CDN. I’d like to know what they are doing.

128MB is more than enough to run Debian and serve a static site. I had no issue with doing it a decade ago and it still works fine.

How much memory do you think it actually takes to accept a TLS connection and copy files from disk to a socket?

Re: Why are anime catgirls blocking my access to the Linux kernel?

#400

Post by pilif »

bananalychee wrote: post #392
pilif wrote: post #389

Earlier quoted context omitted.

> This was obviously dumb when it launched: Yes. Obviously dumb but also nearly 100% successful at the current point in time. And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround. Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources…

Does it actually? I don't think I've seen a case study with hard numbers.

Here’s one study

https://dukespace.lib.duke.edu/server/api/core/bitstreams/81...

And of all the high-profile projects implementing it, like the LKML archives, none have backed down yet, so I’m assuming the initial improvement in numbers must continue or it would have been removed since

Post reply on HN