AWS multiple services outage in us-east-1 • Hacker News Board

Re: AWS multiple services outage in us-east-1

#711

Post by busymom0 » Mon, Oct 20, 2025, 4:49 PM UTC

Forricide wrote: post #703

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Definitely seems to be getting worse, outside of AWS itself, more websites seem to be having sporadic or serious issues. Concerning considering how long the outage has been going.

That's probably why Reddit has been down too

Re: AWS multiple services outage in us-east-1

#712

Post by [deleted] » Mon, Oct 20, 2025, 4:50 PM UTC

ljdtt wrote: post #693

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

[deleted]

Re: AWS multiple services outage in us-east-1

#713

Post by PeterCorless » Mon, Oct 20, 2025, 4:50 PM UTC

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Downdetector had 5,755 reports of AWS problems at 12:52 AM Pacific (3:53 AM Eastern).

That number had dropped to 1,190 by 4:22 AM Pacific (7:22 AM Eastern).

However, that number is back up with a vengeance. 9,230 reports as of 9:32 AM Pacific (12:32 Eastern).

Part of that could be explained by more people making reports as the U.S. west coast awoke. But I also have a feeling that they aren't yet on top of the problem.

Re: AWS multiple services outage in us-east-1

#714

Post by redeux » Mon, Oct 20, 2025, 4:51 PM UTC

It’s a good day to be a DR software company or consultant

Re: AWS multiple services outage in us-east-1

#715

Post by ifwinterco » Mon, Oct 20, 2025, 4:51 PM UTC

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Everything except us-east-1 is generally pretty reliable. At $work we have a lot of stuff that's only on eu-west-1 (yes not the best practice) and we haven't had any issues, touch wood

Re: AWS multiple services outage in us-east-1

#716

Post by outworlder » Mon, Oct 20, 2025, 4:51 PM UTC

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close.

Our company decided years ago to use any region other than us-east-1.

Of course, that doesn't help with services that are 'global', which usually means us-east-1.

Re: AWS multiple services outage in us-east-1

#717

Post by sineausr931 » Mon, Oct 20, 2025, 4:52 PM UTC

On a bright note, Alexa has stopped pushing me merchandise.

Re: AWS multiple services outage in us-east-1

#718

Post by neuroelectron » Mon, Oct 20, 2025, 4:52 PM UTC

Sounds like a circular error with monitoring is flooding their network with metrics and logs, causing DNS to fail and produce more errors, flooding the network. Likely root cause is something like DNS conflicts or hosts being recreated on the network. Generally this is a small amount of network traffic but the LBs are dealing with host address flux, causing the hosts to keep colliding host addresses as they attempt to resolve to a new host address which are being lost from dropped packets and with so many hosts in one AZ, there's a good chance they end up with a new conflicting address.

Re: AWS multiple services outage in us-east-1

#719

Post by czhu12 » Mon, Oct 20, 2025, 4:53 PM UTC

Our entire data stack (Databricks and Omni) are all down for us also. The nice thing is that AWS is so big and widespread that our customers are much more understanding about outages, given that its showing up on the news.

Re: AWS multiple services outage in us-east-1

#720

Post by alchemism » Mon, Oct 20, 2025, 4:54 PM UTC

bigbuppo wrote: post #680

Whose idea was it to make the whole world dependent on us-east-1?

AWS often deploys its new platform products and latest hardware (think GPUs) into us-east-1, so everyone has to maintain a footprint in us-east-1 to use any of it.

So as a result, everyone keeps production in us-east-1. =)