Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

921–930 of 1001 posts

Re: AWS multiple services outage in us-east-1

#921
Anyone needing multi-cloud WITH EASE, please get in touch. https://controlplane.com

I am the CEO of the company and started it because I wanted to give engineering teams an unbreakable cloud. You can mix-n-match services of ANY cloud provider, and workloads failover seamlessly across clouds/on-prem environments.

Feel free to get in touch!

Re: AWS multiple services outage in us-east-1

#922
I wonder what kind of outage or incident or economic change will be required to cause a rejection of the big commercial clouds as the default deployment model.

The costs, performance overhead, and complexity of a modern AWS deployment are insane and so out of line with what most companies should be taking on. But hype + microservices + sunk cost, and here we are.

Re: AWS multiple services outage in us-east-1

#923

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

Wow, you really *have* to exercise the region failover to know if it works, eh? And that confidence gets weaker the longer it’s been since the last failover I imagine too. Thanks for sharing what you learned.

That wouldn't have even caught that, most likely unless they verified they had no incidental tie ins with us-east-1.

Re: AWS multiple services outage in us-east-1

#924

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

Totally ridiculous that AWS wouldn't by default make it multi-region and warn you heavily that your multi-region service is tied to a single region for identity.

The usability of AWS is so poor.

Re: AWS multiple services outage in us-east-1

#925
post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Aren't you deployed in multiple regions?

Re: AWS multiple services outage in us-east-1

#926

I wonder what kind of outage or incident or economic change will be required to cause a rejection of the big commercial clouds as the default deployment model. The costs, performance overhead, and complexity of a modern AWS deployment are insane and so out of line with what most companies should be taking on. But hype + microservices + sunk cost, and here we are.

Honest answer? The outage would need to last about a week.

Re: AWS multiple services outage in us-east-1

#928

It’s that period of the year when we discover AWS clients that don’t have fallback plans

♫ It's the most blunderful time of the year

There'll be much admin moaning

And servers not glowing

and the NOC crew in tears

It's the most blunderful time of the year ♫

Post reply on HN