[deleted]
How to Block Some of the Bots
111–120 of 158 posts
Re: How to Block Some of the Bots
#112Earlier quoted context omitted.
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
There are a lot of urls for the bots to follow. Yesterday's logs had 2.2M unique URLs out of 2.7M requests (more unique than I was expecting!)
Typical traffic:
https://hg.ucc.asn.au/dropbear-fuzzcorpus/annotate/default/fuzzer-preauth/23f6b74342011f35e657f06e9398ad339360d7e9
https://hg.ucc.asn.au/dropbear-fuzzcorpus/comparison/e67e5df225e5/fuzzer-verify/b5f57a2bb85ab7ef957e4979fd202b091de27bd8?revcount=8
https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/72337e3dcccf/fuzzer-preauth_nomaths/6b1f390d26b08759983937727e10ab90aa7b03f2?revcount=15
https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/4cdb539e4b20/fuzzer-preauth_nomaths/713e5cd4f73276f00a5d6685cc16d4565ad3815a?revcount=96
https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/tip/fuzzer-client_nomaths/540778ea81a915af6e898fd9658c5d9d6594c40a?revcount=128
https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/0ef1e51e583f/fuzzer-client/65ac44bdf4868866def70f40542dd1b5ca447310?revcount=30
Re: How to Block Some of the Bots
#113Earlier quoted context omitted.
As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers. I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
Bender, even got yourself a great username. One of the most responsive authors in memory.
Thankyou. It's a fun topic for me. Hopefully others will find it interesting to tinker and experiment with as well.
Re: How to Block Some of the Bots
#114I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
At home, I can't afford a 40gbit connection and a server to match... getting DDoSed by a handful of malicious google cloud VPSes running nmap and various web vulnerability scans can easily impact performance on low end hardware.
Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it, or allow it to continue.
Re: How to Block Some of the Bots
#115Earlier quoted context omitted.
Do you have a source for those numbers?
ChatGPT has over a billion monthly active users.
This is definitely a 2026 type comment, but I'm very surprised that it's so "low." GPT is the most well known brand and it's the one that seemingly 100% of kids use to "assist" on their homework and other exercises from grade school to college. One would think that alone would already take you well over a billion.
Re: How to Block Some of the Bots
#116I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
Bots routinely overwhelm my public-facing websites. The last few times I've turned on "attack mode" in Cloudflare (which I do when bot swarms get particularly nasty), traffic dropped by at least 20k requests/minute (which is huge, considering my human traffic is around 3/4 of that).
Re: How to Block Some of the Bots
#117You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP.
Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.
> One class of bots always claims to come from the root page of google. Obviously this is a lie.
This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
Re: How to Block Some of the Bots
#118Earlier quoted context omitted.
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
> Are these loser companies who don’t want to cache on their end
Google couldn't be bothered to cache; I would guess this before anything else.
https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...
Re: How to Block Some of the Bots
#119Earlier quoted context omitted.
Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially? *I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The…
Didja happen to confirm they were blocked initially They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home inte…
> Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection.
For some reason I find this very amusing.
Re: How to Block Some of the Bots
#120I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…
What about "our hosting bill shot up 500% from bot traffic and went down to the baseline the moment we put our site behind cloudflare or anubis"