I stopped deploying to a single region for production years ago, so I don’t really have a horse in this region comparison race. That said, I’ve seen network level issues in every region I use — nothing like the big outage, but issues that may disrupt a service. Designing for how the world is rather than how I wish it was makes a lot of sense to me.
Is Northern Virginia still the least reliable AWS region?
51–60 of 84 posts
Re: Is Northern Virginia still the least reliable AWS region?
#52There are only two kinds of cloud regions: the ones people complain about and the ones nobody uses
I like this a lot, this is a great comparison for hetzner american offerings since it's not big enough for them to even bother investing much into it so there's not that many complains about it. People just dumping it (me included) after discovering the amount of random issues it has probably also doesn't help.
if you are using hetzner: avoid everything other than fra region, ideally pray that you are placed in the newer part of the datacenter since it has the upgraded switching spine I haven't seen the old one in a bit so they might have deprecated it entirely.
Re: Is Northern Virginia still the least reliable AWS region?
#53There are only two kinds of cloud regions: the ones people complain about and the ones nobody uses
Yeah, I was often the single source of reporting Claude outages (or even missing support completely) on less commonly used Amazon Bedrock regions.
Re: Is Northern Virginia still the least reliable AWS region?
#54There are only two kinds of cloud regions: the ones people complain about and the ones nobody uses
A sound banker, alas, is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way along with his fellows, so that no one can really blame him. JM Keynes
Re: Is Northern Virginia still the least reliable AWS region?
#55Earlier quoted context omitted.
how about following the well-architected framework and building something with a suitable level of 9s where you can justify your decisions during a blameless postmortem (please stamp your buzzword bingo card for a prize.)
We vibe code everything in flavor of the month node frameworks, tyvm, because elixir is too hard to hire for (or some equally inane excuse)
I look forward to the eventual launch of a new and improved version of your app using electron.
What’s the point in having 64 Gb of DDR5 and 16 cores @ 4.2 GHz if not to be able to have a couple electron apps sitting at idle yet somehow still using the equivalent computational resources of the most powerful supercomputer on earth in the mid 1990s.
Re: Is Northern Virginia still the least reliable AWS region?
#56There are only two kinds of cloud regions: the ones people complain about and the ones nobody uses
I like this a lot, this is a great comparison for hetzner american offerings since it's not big enough for them to even bother investing much into it so there's not that many complains about it. People just dumping it (me included) after discovering the amount of random issues it has probably also doesn't help. if you are using hetzner: avoid everything other than fra region, ideally pray that you are placed in the n…
Hetzner does not have any "fra region". They have Helsinki, Falkstien and Nuremberg in Europe. None of them which has any issues as far as I know. They used to have some issues with the very old stuff in Falkstien.
Re: Is Northern Virginia still the least reliable AWS region?
#57There are only two kinds of cloud regions: the ones people complain about and the ones nobody uses
A sound banker, alas, is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way along with his fellows, so that no one can really blame him. JM Keynes
That is incredibly appropriate.
Re: Is Northern Virginia still the least reliable AWS region?
#58Earlier quoted context omitted.
Places nobody's ever heard of like "Ohio" or "Oregon"? Yeah, I'm not worried about being targeted in an RCA and pointedly asked why I chose a region with way better uptime than `us-tirefire-1`. What _is_ worth considering is whether your more carefully considered region will perform better during an actual outage where some critical AWS resource goes down in Virginia, taking my region with it anyway.
IIRC, some AWS services are solely deployed on and/or entirely dependent on us-east-1. I don't recall which ones, but I very distinctly remember this coming up once.
IAM and Route53 have dependencies on us-east-1.
AWS Organizations/Account management is us-east-1.
And if you want a CDN with a custom hostname and want TLS…you have to use us-east-1.
Re: Is Northern Virginia still the least reliable AWS region?
#59I think if you need something more reliable than us-east-1 that you should be hosting on prem in facilities you own and operate.
There aren't that many businesses that truly can't handle the worst case (so far) AWS outage. Payment processing is the strongest example I can come up with that is incompatible with the SLA that a typical cloud provider can offer. Visa going down globally for even a few minutes might be worse than a small town losing its power grid for an entire week.
It's a hell of a lot easier to just go down with everyone else, apologize on Twitter, and enjoy a forced snow day. Don't let it frustrate you. Stay focused on the business and customer experience. It's not ideal to be down, but there are usually much bigger problems to solve. Chasing an extra x% of uptime per year is usually not worth a multicloud/region clusterfuck. These tend to be even less resilient on average.
Re: Is Northern Virginia still the least reliable AWS region?
#60Earlier quoted context omitted.
This to me was the real lesson of the outage. A us-east-1 outage is treated like bad weather. A regional outage can be blamed on the dev. us-east-1 is too big to get blamed, which is why it should be the region of choice for an employee.
Bizarre way of making decisions. us-east-2 is objectively a better region to pick if you want US east, yet you feel safer picking use1 because “I’m safer making a worse decision that everyone understands is worse, as long as everyone else does it as well.”
If my cloud provider goes down and my site is offline, my customers and my boss will be upset with me and demand I fix it as fast as possible. They will not care what caused it.
If my cloud provider goes down and also takes down Spotify, Snapchat, Venmo, Reddit, and a ton of other major services that my customers and my boss use daily, they will be much more understanding that there is a third party issue that we can more or less wait out.
Every provider has outages. US-east-2 will sometimes go down. If I'm not going to make a system that can fail over from one provider to another (which is a lot of work and can be expensive, and really won't be actively used often), it might be better to just use the popular one and go with the group.