Looks like Anubis is also blocking robots.txt which seems to defeat the point of having robots.txt in the first place.
The bot situation on the internet is worse than you could imagine
31–40 of 179 posts
Re: The bot situation on the internet is worse than you could imagine
#32This is why I see (well managed) government digital IDs as sensible moves. Apart from DDOS attacks, if bots have to “prove” who they are on each request it seems like a win-win. I may be missing something of course
If you want “papers, please” every time you back out of your driveway or go beyond your government-assigned oblast, then your suggestion is the digital version of the physical authoritarian nightmare that was imposed by totalitarianist regimes throughout history.
People have a right to complete anonymity, and should be able to go across the majority of the Internet just as they can go across most of the country.
That’s what you are missing.
Don’t get me wrong, I am also in favour of a single government ID, but in terms of combatting identity fraud, accessing public resources like single-payer healthcare, and making it easier for a person to prove their identity to authorities or employers.
It should not be used as a pass card for fundamental rights that normally would have zero government involvement.
Re: The bot situation on the internet is worse than you could imagine
#33I'm surprised at the effectiveness of simple PoW to stop practically all activity. I'll implement Anubis at low difficulty for all my projects and leave a decent llms.txt referenced in my sitemap and robots.txt so LLMs can still get relevant data for my site while.keeping bad bots out. I'm getting thousands of requests from China that have really increased costs, glad it seems the fix is rather easy.
>I'm surprised at the effectiveness of simple PoW to stop practically all activity. It's even dumber than that, because by default anubis whitelists the curl user agent. curl -H "User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/diff/?id=v7.0-rc5&id2=v7.0-rc4&dt=2" Making sure…
shhhh don't tell the bots !
Re: The bot situation on the internet is worse than you could imagine
#34I'm surprised at the effectiveness of simple PoW to stop practically all activity. I'll implement Anubis at low difficulty for all my projects and leave a decent llms.txt referenced in my sitemap and robots.txt so LLMs can still get relevant data for my site while.keeping bad bots out. I'm getting thousands of requests from China that have really increased costs, glad it seems the fix is rather easy.
>I'm surprised at the effectiveness of simple PoW to stop practically all activity. It's even dumber than that, because by default anubis whitelists the curl user agent. curl -H "User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/diff/?id=v7.0-rc5&id2=v7.0-rc4&dt=2" Making sure…
Anubis' white lists and block rules are configurable though. The defaults are a bit silly.
Re: The bot situation on the internet is worse than you could imagine
#35> Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours I love experimental data like this. So much better than gut reaction that was spammed when anubis was just introduced
Well yeah, but I also didn't make it through to the actual site. That can't be the idea, right? After 5 seconds of 100% CPU and no progress I gave up.
The idea is to scare off bots and not normal humans.
Re: The bot situation on the internet is worse than you could imagine
#36Is Anubus being set to difficulty 8 on this page supposed to be a joke? I gave up after about 20 seconds.
I waited a minute until my phone got hot.
Re: The bot situation on the internet is worse than you could imagine
#37I’ve been sitting on this page for two minutes and it’s still not sure whether I’m a bot lol. What did I do in a past life to deserve this :(
After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
I don't get this kHash thing. Do we have captchas mining bitcoin in a distributed fashion for free now?
Re: The bot situation on the internet is worse than you could imagine
#38I’ve been sitting on this page for two minutes and it’s still not sure whether I’m a bot lol. What did I do in a past life to deserve this :(
After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScript which is what the Anubis tool does.
for (; ;) {
const hashBuffer = await calculateSHA256(data + nonce);
const hashArray = new Uint8Array(hashBuffer);
let isValid = true;
for (let i = 0; i Re: The bot situation on the internet is worse than you could imagine
#39[deleted]
Re: The bot situation on the internet is worse than you could imagine
#40I’ve been sitting on this page for two minutes and it’s still not sure whether I’m a bot lol. What did I do in a past life to deserve this :(
After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
At this point I wonder if you can post a crypto miner page on HN and people will fall for it.