Anthem Health call center disconnected my wife numerous times yesterday with an ominous robo-message of "Emergency in our call center"; curious if that was this. Seems likely, but what a weird message.
AWS multiple services outage in us-east-1
971–980 of 1001 posts
Re: AWS multiple services outage in us-east-1
#972Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…
People will continue to purchase Mutli-AZ and multi-region even though you have proved what a scam it is. If east region goes down, ALL amazon goes down, feel free to change my mind. STOP paying double rates for multi region.
Re: AWS multiple services outage in us-east-1
#973Do we know what caused the outage yet?
Re: AWS multiple services outage in us-east-1
#974this is why you avoid us-east-1
And yet you're still impacted because it hosts IAM
Re: AWS multiple services outage in us-east-1
#975Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!
This was a single region outage, right? If you aren't cross-region, cross-cloud is the same but harder
Re: AWS multiple services outage in us-east-1
#976Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!
Is us-east-1 equally unstable to the other regions? My impression was that Amazon deployed changes to us-east-1 first so it's the most unstable region.
Would think that Amazon safeguards their biggest region more, but no idea, I've never worked at AWS
Re: AWS multiple services outage in us-east-1
#977Are there websites that do post-mortems for how the single points of failure impacted the entire internet?
Not just AWS, but Cloudflare and others too. Would be interesting to review them clinically.
Re: AWS multiple services outage in us-east-1
#978Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our company decided years ago to use any region other than us-east-1. Of course, that doesn't help with services that are 'global', which usually means us-east-1.
One advantage to being in the biggest region: when it goes down the headlines all blame AWS, not you. Sure you’re down too, but absolutely everybody knows why and few think it’s your fault.
Re: AWS multiple services outage in us-east-1
#979Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!
I was once told that our company went with Azure because when you tell the boomer client that our service is down because Microsoft had an outage, they go from being mad at you, to accepting that the outage was an act of god that couldn’t be avoided.
Azure outages: happens all the time, understandable, no way to prevent this
AWS outages: almost never happens, you should have been more prepared for when it does
Re: AWS multiple services outage in us-east-1
#980As Amazon moves from day-1 company as it claimed once, to be the sales company like Oracle focusing on raking money, expect more outages to come, and longer to be resolved. Amazon is burning and driving away the technical talent and knowledge knowing the vendor lock-in will keep bringing the sweet money. You will see more sales people hoovering around your c-suites and executives, while you will face even worse techn…
https://www.theregister.com/2025/10/20/aws_outage_amazon_bra... quoting from this "And so, a quiet suspicion starts to circulate: where have the senior AWS engineers who've been to this dance before gone? And the answer increasingly is that they've left the building — taking decades of hard-won institutional knowledge about how AWS's systems work at scale right along with them." ... "AWS has given increasing levels o…
[deleted]