AWS multiple services outage in us-east-1 • Hacker News Board

Re: AWS multiple services outage in us-east-1

#771

Post by ttul » Mon, Oct 20, 2025, 5:42 PM UTC

ifwinterco wrote: post #715

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Everything except us-east-1 is generally pretty reliable. At $work we have a lot of stuff that's only on eu-west-1 (yes not the best practice) and we haven't had any issues, touch wood

My impression is that `us-east-1` has the worst reliability track record of any region. We've always run our stuff in `us-west-2` and there has never been an outage that took us down in that region. By contrast, a few things that we had in `us-east-1` have gone down repeatedly.

Re: AWS multiple services outage in us-east-1

#772

Post by rdtsc » Mon, Oct 20, 2025, 5:44 PM UTC

markus_zhang wrote: post #743

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

It has been quite a while, wondering how many 9s are dropped. 365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.

9s don’t have to drop if you increase the time period! “We still guarantee the same 9s just over 3450 years now”.

Re: AWS multiple services outage in us-east-1

#773

Post by BoredPositron » Mon, Oct 20, 2025, 5:45 PM UTC

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Still no serverless inference for models or inference pipes that are not available on bedrock, still no auto scaling GPU workers. We started bothering them in 2022...crickets

Re: AWS multiple services outage in us-east-1

#774

Post by mcintyre1994 » Mon, Oct 20, 2025, 5:46 PM UTC

stego-tech wrote: post #758

Not remotely surprised. Any competent engineer knows full well the risk of deploying into us-east-1 (or any “default” region for that matter), as well as the risks of relying on global services whose management or interaction layer only exists in said zone. Unfortunately, us-east-1 is the location most outsourcing firms throw stuff, because they don’t have to support it when it goes pear-shaped (that’s the client’s p…

> as well as the risks of relying on global services whose management or interaction layer only exists in said zone.

Is this well known/documented? I don't have anything on AWS but previously worked for a company that used it fairly heavily. We had everything in EU regions and I never saw any indication/warning that we had a dependency on us-east-1. But I assume we probably did based on the blast radius of today's outage.

Re: AWS multiple services outage in us-east-1

#775

Post by iLoveOncall » Mon, Oct 20, 2025, 5:46 PM UTC

everfrustrated wrote: post #754

Waterluvian wrote: post #695

I know there's a lot of anecdotal evidence and some fairly clear explanations for why `us-east-1` can be less reliable. But are there any empirical studies that demonstrate this? Like if I wanted to back up this assumption/claim with data, is there a good link for that, showing that us-east-1 is down a lot more often?

The unreliability claim is driven by two factors. 1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. Most services start with us-east-1 first. 2. us-east-1 is MASSIVE and considerably larger than the next largest region. There's no public numbers but I wouldn't be surprised if it was 50% of their global capacity. An outage in any other region never hits the news.

> 1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time.

This is true.

> Most services start with us-east-1 first.

This is absolutely false. Almost every service will FINISH with the largest and most impactful regions.

Re: AWS multiple services outage in us-east-1

#776

Post by mvdtnz » Mon, Oct 20, 2025, 5:49 PM UTC

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

The problems now seem mostly related to starting new instances. Our capacity is slowly decaying as existing services spin down and new EC2 workloads fail to start.

Re: AWS multiple services outage in us-east-1

#777

Post by melozo » Mon, Oct 20, 2025, 5:49 PM UTC

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

Re: AWS multiple services outage in us-east-1

#778

Post by Bolwin » Mon, Oct 20, 2025, 5:52 PM UTC

wcchandler wrote: post #742

This is usually something I see on Reddit first, within minutes. I’ve barely seen anything on my front page. While I understand it’s likely the subs I’m subscribed to, that was my only reason for using Reddit. I’ve noticed that for the past year - more and more tech heavy news events don’t bubble up as quickly anymore. I also didn’t see this post for a while for whatever reason. And Digg was hit and miss on availabil…

Try lemmy, it's currently in the front page, albeit on a meme sub

Re: AWS multiple services outage in us-east-1

#779

Post by steveBK123 » Mon, Oct 20, 2025, 5:55 PM UTC

altbdoor wrote: post #678

Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS service. I asked if it was a single point of failure. The whole room laughed, I rest my case.

I mean given what we've seen with these AWS failures impact, wouldn't any enemies first target be to hit us-east-1 ? Imagine if it just disappeared?

Re: AWS multiple services outage in us-east-1

#780

Post by loudmax » Mon, Oct 20, 2025, 5:56 PM UTC

whaleofatw2022 wrote: post #694

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Dangerous curiosity ask, is whether the number of folks off for Diwali is a factor or not? I.e. lots of folks that weren't expected to work today and/or trying to round them up to work the problem.

Northern Virginia's Fairfax County public schools have the day off for Diwali, so that's not an unreasonable question.

In my experience, the teams at AWS are pretty diverse, reflecting the diversity in the area. Even if a lot of the Indian employees are taking the day off, there should be plenty of other employees to back them up. A culturally diverse employee base should mitigate against this sort of problem.

If it does turn out that the outage was prolonged due to one or two key engineers being unreachable for the holiday, that's an indictment of AWS for allowing these single points of failure to occur, not for hiring Indians.