Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are anime catgirls blocking my access to the Linux kernel?

lock.cmpxchg8b.com

Post reply on HN
Print view View original

541–550 of 968 posts

Re: Why are anime catgirls blocking my access to the Linux kernel?

#541

Post by topranks »

johnklos wrote: post #244

This is a usually technical crowd, so I can't help but wonder if many people genuinely don't get it, or if they are just feigning a lack of understanding to be dismissive of Anubis. Sure, the people who make the AI scraper bots are going to figure out how to actually do the work. The point is that they hadn't, and this worked for quite a while. As the botmakers circumvent, new methods of proof-of-notbot will be made…

Sure.

It might be a tool in the box. But it’s still cat and mouse.

In my place we quickly concluded the scrapers have tons of compute and the “proof-of-work” aspect was meaningless to them. It’s simply the “response from site changed, need to change our scraping code” aspect that helps.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#542

Post by elsjaako »

zaptrem wrote: post #531

If people are truly concerned about the crawlers hammering their 128mb raspberry pi website then a better solution would be to provide an alternative way for scrapers to access the data (e.g., voluntarily contribute a copy of their public site to something like common crawl). If Anubis blocked crawler requests but helpfully redirected to a giant tar ball of every site using their service (with deltas or something to…

There's a lot of people that really don't like AI, and simply don't want their data used for it.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#543

Post by sussmannbaka »

zaptrem wrote: post #531

If people are truly concerned about the crawlers hammering their 128mb raspberry pi website then a better solution would be to provide an alternative way for scrapers to access the data (e.g., voluntarily contribute a copy of their public site to something like common crawl). If Anubis blocked crawler requests but helpfully redirected to a giant tar ball of every site using their service (with deltas or something to…

No, this doesn’t work. Many of the affected sites have these but they’re ignored. We’re talking about git forges, arguably the most standardised tool in the industry, where instead of just fetching the repository every single history revision of every single file gets recursively hammered to death. The people spending the VC cash to make the internet unusable right now don’t know how to program. They especially don’t give a shit about being respectful. They just hammer all the sites, all the time, forever.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#544

Post by wongarsu »

mlyle wrote: post #379
technion wrote: post #362

Earlier quoted context omitted.

It really should be recognised just how many people are watching Cloudflare interstitials on nearly every site these days (and I totally get why this happens) yet making a huge amount of noise about Anubis on a very small amount of sites.

I don't trip over CloudFlare except when in a weird VPN, and then it always gets out of my way after the challenge. Anubis screws with me a lot, and often doesn't work.

For me both are things that mostly show up for 1-3 seconds, then get replaced by the actual website. I suspect that's the user experience of 99% of people.

If you fall in the other 1% (e.g. due to using unusual browsers or specific IP ranges), cloudflare tends to be much worse

Re: Why are anime catgirls blocking my access to the Linux kernel?

#545

Post by alibarber »

Pinus wrote: post #463
mlyle wrote: post #379

Earlier quoted context omitted.

I don't trip over CloudFlare except when in a weird VPN, and then it always gets out of my way after the challenge. Anubis screws with me a lot, and often doesn't work.

I’m planning a trip to France right now, and it seems like half the websites in that country (for example, ratp.fr for Paris public transport info) require me to check a CloudFlare checkbox to promise that I am a human. And of those that don’t, quite a few just plain lock me out...

It's not hard to understand why though surely?

You might have to show a passport when you enter France, and have your baggage and person (intrusively) scanned if you fly there, for much the same reason.

People, some of them in positions of government in some nation states want to cause harm to the services of other states. Cloudflare was probably the easiest tradeoff for balancing security of the service with accessibility and cost to the French/Parisian taxpayer.

Not that I'm happy about any of this, but I can understand it.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#546

Post by Symbiote »

immibis wrote: post #35
Philpax wrote: post #21

Earlier quoted context omitted.

Volume, primarily - the scrapers are running full-tilt, which many dynamic websites aren't designed to handle: https://pod.geraspora.de/posts/17342163

Why haven't they been sued and jailed for DDoS, which is a felony?

Many are using botnets, so it's not practical to find out who they are.

Re: Why are anime catgirls blocking my access to the Linux kernel?

#547

Post by lmm »

zaptrem wrote: post #531

If people are truly concerned about the crawlers hammering their 128mb raspberry pi website then a better solution would be to provide an alternative way for scrapers to access the data (e.g., voluntarily contribute a copy of their public site to something like common crawl). If Anubis blocked crawler requests but helpfully redirected to a giant tar ball of every site using their service (with deltas or something to…

The kind of crawlers/scrapers who DDoS a site like this aren't going to bother checking common crawl or tarballs. You vastly overestimate the intelligence and prosociality of what bursty crawler requests tend to look like. (Anyone who is smart or prosocial will set up their crawler to not overwhelm a site with requests in the first place - yet any site with any kind of popularity gets flooded with these requests sooner or later)

Re: Why are anime catgirls blocking my access to the Linux kernel?

#549

Post by zaptrem »

elsjaako wrote: post #542
zaptrem wrote: post #531

If people are truly concerned about the crawlers hammering their 128mb raspberry pi website then a better solution would be to provide an alternative way for scrapers to access the data (e.g., voluntarily contribute a copy of their public site to something like common crawl). If Anubis blocked crawler requests but helpfully redirected to a giant tar ball of every site using their service (with deltas or something to…

There's a lot of people that really don't like AI, and simply don't want their data used for it.

While that’s a reasonable opinion to have, it’s a fight they can’t really win. It’s like putting up a poster in a public square then running up to random people and shouting “no, this poster isn’t for you because I don’t like you, no looking!” Except the person they’re blocking is an unstoppable mega corporation that’s not even morally in the wrong imo (except for when they overburden people’s sites, that’s bad ofc)

Re: Why are anime catgirls blocking my access to the Linux kernel?

#550

Post by guappa »

vmttmv wrote: post #234
thunderfork wrote: post #185

Earlier quoted context omitted.

Once you've built the system that lets you do that, maybe. You still have to do that, though, so it's still raising the cost floor.

but... how? when the author ran the numbers, the rough estimate is solving the challenges at a rate of 10000/5 min, on a single instance of the free tier of google compute. that is an insignificant load at an even more insignificant cost.

That's insanely slow compared to how fast they normally scrape.
Post reply on HN