Major us-east-1 outages happened in 2011, 2015, 2017, 2020, 2021, 2023, and now again. I understand that us-east-1, N. VA, was the first DC but for fucks sake they've had HOW LONG to finish AWS and make us-east-1 not be tied to keeping AWS up.
AWS multiple services outage in us-east-1
381–390 of 1001 posts
Re: AWS multiple services outage in us-east-1
#382Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…
Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.
My website running on an old laptop in my cupboard is doing just fine.
Re: AWS multiple services outage in us-east-1
#383Reddit seems to be having issues too:
"upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout"
Re: AWS multiple services outage in us-east-1
#384I like that we can advertise to our customers that over the last X years we have better uptime than Amazon, google, etc.
The Register calls it Microsoft 364, 363, ...
Re: AWS multiple services outage in us-east-1
#385Seems to have taken down my router "smart wifi" login page, and there's no backup router-only login option! Brilliant work, linksys....
Re: AWS multiple services outage in us-east-1
#386I like that we can advertise to our customers that over the last X years we have better uptime than Amazon, google, etc.
Reported uptimes are little more than fabricated bullshit.
They measure uptime using averages of "if any part of a chain is even marginally working".
People experience downtime however as "if any part of a chain is degraded".
Re: AWS multiple services outage in us-east-1
#387Is this why reddit is down? ( https://www.redditstatus.com/ still says it is up but with degraded infrastructure)
Shameless from them to make it look like it's a user problem.
It was loading fine for me one hour ago, now I refresh the page and their message states I'm doing too many requests and should chill out (1 request per hour is too many for you?)
Re: AWS multiple services outage in us-east-1
#388Wasn't the point why AWS is so much premium that you will always get at least 6 nines if not more in availability?
The point of AWS is to promise you the nines and make you feel good about it. Your typical "growth & engagement" startup CEO can feel good and make his own customers feel good about how his startup will survive a nuclear war.
Delivery of those nines is not a priority. Not for the cloud provider - because they can just lie their way out of it by not updating their status page - and even when they don't, they merely have to forego some of their insane profit margin for a couple hours in compensation. No provider will actually put their ass on the line and offer you anything beyond their own profit margin.
This is not an issue for most cloud clients either because they keep putting up with it (lying on the status page wouldn't be a thing if clients cared) - the unspoken truth is that nobody cares that your "growth & engagement" thing is down for an hour or so, so nobody makes anything more than a ceremonial stink about it (chances are, the thing goes down/misbehaves regularly anyway every time the new JS vibecoder or "AI employee" deploys something, regardless of cloud reliability).
Things where nines actually matter will generally invest in self-managed disaster recovery plans that are regularly tested. This also means it will generally be built differently and far away from your typical "cloud native" dumpster fire. Depending on how many nines you actually need (aka what's the cost of not meeting that target - which directly controls how much budget you have to ensure you always meet it), you might be building something closer to aircraft avionics with the same development practices, testing and rigor.
Re: AWS multiple services outage in us-east-1
#389It’s that period of the year when we discover AWS clients that don’t have fallback plans
Re: AWS multiple services outage in us-east-1
#390Major us-east-1 outages happened in 2011, 2015, 2017, 2020, 2021, 2023, and now again. I understand that us-east-1, N. VA, was the first DC but for fucks sake they've had HOW LONG to finish AWS and make us-east-1 not be tied to keeping AWS up.
First, not all outages are created equal, so you cannot compare them like that.
I believe the 2021 one was especially horrific because of it affecting their dns service (route53) and the outage made writes to that service impossible. This made fail overs not work etcetera so their prescribed multi region setups didn't work.
But in the end, some things will have to synchronizes their writes somewhere, right? So for dns I could see how that ends up in a single region.
AWS is bound by the same rules as everyone else in the end... The only thing they have going for them that they have a lot of money to make certain services resilient, but I'm not aware of a single system that's resilient to everything.