So that’s why bot attacks and spam traffic was lower.
DigitalOcean block storage is down
71–80 of 88 posts
Re: DigitalOcean block storage is down
#72Earlier quoted context omitted.
Yup this is normal, when I can't change the SSH port (e.g for compatibility) then I switch f2b to permanently ban, which should reduce any incurred load by black-holing instead of attempting authentication as the list grows more comprehensive over time (this wont affect other services e.g apache, in-case a user is unwittingly part of a botnet).
Fatfingered typist beware...
Better use a password manager in this case (or even better, use public key auth!)
Re: DigitalOcean block storage is down
#73Earlier quoted context omitted.
to be fair, it's dangerous even in the best hands. mistakes happen but business processes need to be in place to prevent catastrophes... every time i see something like this, my inclination is to blame the CTO, not the engineer who pulled the trigger.
We can blame whoever we want but you better believe shit rolls downhill at most places.
Until it is a big enough F-up that an executive's head must roll.
Re: DigitalOcean block storage is down
#74DigitalOCean bad experience
Re: DigitalOcean block storage is down
#75This is OT, but I have a droplet on DO and I'm amazed at the amount of malicious traffic it gets. Is it normal for a very private vps to receive thousands of ssh attempts per hour? I have fail2ban installed and the jail is so busy it's quite astounding. Anyone with more web hosting experience that can weigh in?
> Is it normal for a very private vps to receive thousands of ssh attempts per hour? Yes. The thing is about the IPv4 space is that it’s really not that big (3,706,452,992 public addresses) so it’s pretty trivial to poke every single one esp if you fine tune your port list. The most common advice is to hide your private services. Instead of using port 22 for ssh use 23231 instead. It’s a little more annoying but you…
Another really good option is single packet authorization. Which, again, a little more complex than port knocking but also another step up in security.
Re: DigitalOcean block storage is down
#76Re: DigitalOcean block storage is down
#77Earlier quoted context omitted.
That would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.
Those are not mutually exclusive and actually a good idea. You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. That's scaling and redundancy 101 - not sure why it would be something wrong.
> Those are not mutually exclusive and actually a good idea.
The goals “assuring folks that there is isolation” and “identifying flaws that need to be fixed” are somewhat contrary to each other.
The post-mortem should identify flaws in systems, processes, and thinking. It should not try to assure people that there is isolation when there is evidence to the contrary.
> You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs.
This was a multi-regional failure. So, this specific issue is also an isolation problem, among other things. You will want to ensure that this problem doesn’t happen again but you shouldn’t assure that it won’t.
Re: DigitalOcean block storage is down
#78Earlier quoted context omitted.
I work for a web hosting company in Texas, and this is ridiculously common. Any public IP with any public service at all will be poked, prodded, and generally made uncomfortable by every bot and crawler you can think of, trying common password combinations and scanning for common vulnerabilities in popular software. This catches so many of our customers by surprise, who tend to mistakenly believe they're being target…
I always switch my outward-facing ssh servers to key-only. Is there any advantage for running fail2ban additionally?
for my DO droplet I also changed the ssh port to a silly-high random port and the last time I checked it reduced the amount of nosy bots knocking at the door to zero.
Re: DigitalOcean block storage is down
#79Earlier quoted context omitted.
I always switch my outward-facing ssh servers to key-only. Is there any advantage for running fail2ban additionally?
for my DO droplet I also changed the ssh port to a silly-high random port and the last time I checked it reduced the amount of nosy bots knocking at the door to zero.
I used to do so too, but sometimes had problems with very restrictive firewalls killing connections to high/unknown ports when traveling. They would only allow vpns or ssh to connect.
Re: DigitalOcean block storage is down
#80Earlier quoted context omitted.
That would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.
I would think having all zones go down is a flaw that genuinely needed to be fixed.
That’s not the flaw, that’s the outcome. The purpose of a post-mortem is to identify the flaws that caused that outcome, and ways to fix those flaws.