AWS multiple services outage in us-east-1 • Hacker News Board

Re: AWS multiple services outage in us-east-1

#681

Post by redwood » Mon, Oct 20, 2025, 4:38 PM UTC

Surprising and sad to see how many folks are using DynamoDB There are more full featured multi-cloud options that don't lock you in and that don't have the single point of failure problems.

And they give you a much better developer experience...

Sigh

Re: AWS multiple services outage in us-east-1

#682

Post by pardner » Mon, Oct 20, 2025, 4:38 PM UTC

Darn, on Heroku even the "maintenance mode" (redirects all routes to a static url) won't kick in.

Re: AWS multiple services outage in us-east-1

#683

Post by muttantt » Mon, Oct 20, 2025, 4:38 PM UTC

mannyv wrote: post #603

This is why we use us-east-2.

us-east-2 remains the best kept secret

Re: AWS multiple services outage in us-east-1

#684

Post by autophagian » Mon, Oct 20, 2025, 4:39 PM UTC

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Yeah. We had a brief window where everything resolved and worked and now we're running into really mysterious flakey networking issues where pods in our EKS clusters timeout talking to the k8s API.

Re: AWS multiple services outage in us-east-1

#685

Post by [deleted] » Mon, Oct 20, 2025, 4:39 PM UTC

[deleted]

Re: AWS multiple services outage in us-east-1

#686

Post by caymanjim » Mon, Oct 20, 2025, 4:40 PM UTC

GoblinSlayer wrote: post #393

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

But then you will be affected by outages of every dependency you use.

This is the real problem. Even if you don't run anything in AWS directly, something you integrate with will. And when us-east-1 is down, it doesn't matter if those services are in other availability zones. AWS's own internal services rely heavily on us-east-1, and most third-party services live in us-east-1.

It really is a single point of failure for the majority of the Internet.

Re: AWS multiple services outage in us-east-1

#687

Post by JCM9 » Mon, Oct 20, 2025, 4:40 PM UTC

At 3:03 AM PT AWS posted that things are recovering and sounded like issue was resolved.

Then things got worse. At 9:13 AM PT it sounds like they’re back to troubleshooting.

Honestly sounds like AWS doesn’t even really know what’s going on. Not good.

Re: AWS multiple services outage in us-east-1

#688

Post by nemomarx » Mon, Oct 20, 2025, 4:41 PM UTC

bigbuppo wrote: post #680

Whose idea was it to make the whole world dependent on us-east-1?

The NSA might be happy everything runs through a local data center to their Virginia offices

Re: AWS multiple services outage in us-east-1

#689

Post by JCM9 » Mon, Oct 20, 2025, 4:41 PM UTC

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Agree… still seeing major issues. Briefly looked like it was getting better but things falling apart again.

Re: AWS multiple services outage in us-east-1

#690

Post by lenerdenator » Mon, Oct 20, 2025, 4:41 PM UTC

bigbuppo wrote: post #680

Whose idea was it to make the whole world dependent on us-east-1?

1) People in us-east-1.

2) People who thought that just having stuff "in the cloud" meant that it was automatically spread across regions. Hint, it's not; you have to deploy it in different regions and architect/maintain around that.

3) Accounting.