Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

501–510 of 704 posts

Re: Creepy Crawlies

#501

Post by akerl_ »

kstrauser wrote: post #484
tptacek wrote: post #67

Earlier quoted context omitted.

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in…

Have you compared against adding a JavaScript requirement with no proof of work?

Re: Creepy Crawlies

#502

Post by charcircuit »

rcxdude wrote: post #38
chuckadams wrote: post #8

Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.

The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'

Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.

Re: Creepy Crawlies

#503

Post by FergusArgyll »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Woah, went from > 1 min on git.ffmpeg (difficulty 6) to instant. Thanks!

Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds

Re: Creepy Crawlies

#504

Post by Dylan16807 »

fc417fc802 wrote: post #461
chr15m wrote: post #426

Earlier quoted context omitted.

Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.

> Anubis wants "slow for bots but fast for humans" That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.

That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance.

I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?

Re: Creepy Crawlies

#505

Post by kstrauser »

akerl_ wrote: post #501
kstrauser wrote: post #484

Earlier quoted context omitted.

And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in…

Have you compared against adding a JavaScript requirement with no proof of work?

Yes. That helped, but not nearly so much.

Re: Creepy Crawlies

#506

Post by Dylan16807 »

latentsea wrote: post #445
gizmodo59 wrote: post #342

Earlier quoted context omitted.

Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.

It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.

"It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems."

I don't see much connection between that problem and these controls.

Re: Creepy Crawlies

#507

Post by lmz »

phyzome wrote: post #470
lmz wrote: post #418

It's funny how some people say "AI bad, datacenters waste energy" then other people say "AI bad, going to make humans and their phones waste energy".

This is a silly take. AI is why they're having to waste energy.

Sure, and the humans and their phones are fair collateral damage in that worldview.

Re: Creepy Crawlies

#509

Post by inigyou »

leoqa wrote: post #385
inigyou wrote: post #295

Earlier quoted context omitted.

In fact they already do this. Buying 100 android phones and chargers is cheaper than reverse engineering whatever you're trying to automate - or was, before AI.

.. and we ban those device ids and move on. Your capital is lost.

Evidence shows otherwise. There are people making lots of money from these device farms. Right now. This isn't hypothetical.

Re: Creepy Crawlies

#510

Post by inigyou »

TZubiri wrote: post #468
inigyou wrote: post #294

Earlier quoted context omitted.

Cloudflare doesn't block bots.

That's the whole raison d'etre for CloudFlare, it was originally a DDoS protection layer, which, as the article mentions, is the final form of malicious traffic, being distributed and hard to attribute traffic to an identity. If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.

That's what their marketing tells you it does - not what it actually does.
Post reply on HN