Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

341–350 of 683 posts

Re: Creepy Crawlies

#341

Post by samtheDamned »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.

Re: Creepy Crawlies

#342

Post by gizmodo59 »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.

Re: Creepy Crawlies

#345

Post by ButlerianJihad »

mbirth wrote: post #338
inigyou wrote: post #268

Earlier quoted context omitted.

abuseipdb.com Some ISPs ban customers based on a single report there - have fun!

Many thanks! I've just submitted the first batch of 3000 (daily limit) IP addresses.

That is a ridiculous way to try and deal with the problem of residential proxies.

You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".

If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.

You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.

Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."

Re: Creepy Crawlies

#346

Post by zalo »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...

Re: Creepy Crawlies

#347

Post by colinsane »

KomoD wrote: post #133
yellow_lead wrote: post #25

High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

> High Anubis difficulty is annoying the hell out of me for several sites. I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.

the brutal truth is that if the website operator simply disabled Anubis, your page load would likely take more than 30s.

when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.

30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.

Re: Creepy Crawlies

#348

Post by andy81 »

MithrilTuxedo wrote: post #68
yellow_lead wrote: post #25

High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

Has anyone considered having Anubis perform more valuable hashing? Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pick…

https://en.wikipedia.org/wiki/Hashcash

Re: Creepy Crawlies

#349

Post by calvinmorrison »

riknos314 wrote: post #320
calvinmorrison wrote: post #249

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate. The following from Claude: """ A diff between two randomly chosen commits usua…

cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.

Re: Creepy Crawlies

#350

Post by semiquaver »

gizmodo59 wrote: post #342
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.

I used sol as well. Should have noted that fable is more or less guaranteed to refuse something hacking-adjacent like that.
Post reply on HN