AWS multiple services outage in us-east-1 • Hacker News Board

Re: AWS multiple services outage in us-east-1

#931

Post by babl-yc » Tue, Oct 21, 2025, 3:55 AM UTC

rsanheim wrote: post #922

I wonder what kind of outage or incident or economic change will be required to cause a rejection of the big commercial clouds as the default deployment model. The costs, performance overhead, and complexity of a modern AWS deployment are insane and so out of line with what most companies should be taking on. But hype + microservices + sunk cost, and here we are.

I don't expect the majority of tech companies to want to run their own physical data centers. I do expect them to shift to more bare-metal offerings.

If I'm a mid to large size company built on DynamoDB, I'd be questioning if it's really worth the risk given this 12+ hour outage.

I'd rather build upon open source tooling on bare metal instances and control my own destiny, than hope that Amazon doesn't break things as they scale to serve a database to host the entire internet.

For big companies, it's probably a cost savings too.

Re: AWS multiple services outage in us-east-1

#932

Post by reenorap » Tue, Oct 21, 2025, 4:28 AM UTC

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

It's a good reminder actually that if you don't test the failover process, you have no failover process. The CTO or VP of Engineering should be held accountable for not making sure that the failover process is tested multiple times a month and should be seamless.

Re: AWS multiple services outage in us-east-1

#933

Post by Gigachad » Tue, Oct 21, 2025, 5:08 AM UTC

nikolay wrote: post #46

Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!

I was once told that our company went with Azure because when you tell the boomer client that our service is down because Microsoft had an outage, they go from being mad at you, to accepting that the outage was an act of god that couldn’t be avoided.

Re: AWS multiple services outage in us-east-1

#934

Post by sroussey » Tue, Oct 21, 2025, 5:08 AM UTC

reenorap wrote: post #932

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

It's a good reminder actually that if you don't test the failover process, you have no failover process. The CTO or VP of Engineering should be held accountable for not making sure that the failover process is tested multiple times a month and should be seamless.

If you don’t regularly restore a backup, you don’t have one.

Re: AWS multiple services outage in us-east-1

#935

Post by FlynnLivesMattr » Tue, Oct 21, 2025, 5:25 AM UTC

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

How did the call go?

Re: AWS multiple services outage in us-east-1

#936

Post by Eridrus » Tue, Oct 21, 2025, 5:29 AM UTC

outworlder wrote: post #716

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our company decided years ago to use any region other than us-east-1. Of course, that doesn't help with services that are 'global', which usually means us-east-1.

This was a major issue, but it wasn't a total failure of the region.

Our stuff is all in us-east-1, ops was a total shitshow today (mostly because many 3rd party services besides aws were down/slow), but our prod service was largely "ok", a total of I think we got a bit lucky, but no actual SLAs were violated. I tagged the postmortem as Low impact despite the stress this caused internally.

We definitely learnt something here about both our software and our 3rd party dependencies.

Re: AWS multiple services outage in us-east-1

#937

Post by kelseydh » Tue, Oct 21, 2025, 5:46 AM UTC

nikolay wrote: post #46

Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!

Is us-east-1 equally unstable to the other regions? My impression was that Amazon deployed changes to us-east-1 first so it's the most unstable region.

Re: AWS multiple services outage in us-east-1

#938

Post by ct520 » Tue, Oct 21, 2025, 5:56 AM UTC

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

I always find it interesting how many large enterprises have all these DR guidelines but fail to ever test. Glad to hear that everything came back alright

Re: AWS multiple services outage in us-east-1

#939

Post by bean469 » Tue, Oct 21, 2025, 5:57 AM UTC

radium3d wrote: post #827

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.

> What are you going to do? Host in house?

Yep. Although it's just anecdata, it's what we do where I work - haven't had a slightest issue in years.

Re: AWS multiple services outage in us-east-1

#940

Post by fastball » Tue, Oct 21, 2025, 6:16 AM UTC

One of my co-workers was woken up by his Eight Sleep going haywire. He couldn't turn it off because the app wouldn't work (presumably running on AWS).