Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: AWS multiple services outage in us-east-1

#691

Post by vladvasiliu »

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

> Identity Center and only put it in us-east-1

Is it possible to have it in multiple regions? Last I checked, it only accepted one region. You needed to remove it first if you wanted to move it.

Re: AWS multiple services outage in us-east-1

#693

Post by ljdtt »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

Re: AWS multiple services outage in us-east-1

#694

Post by whaleofatw2022 »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Dangerous curiosity ask, is whether the number of folks off for Diwali is a factor or not?

I.e. lots of folks that weren't expected to work today and/or trying to round them up to work the problem.

Re: AWS multiple services outage in us-east-1

#695

Post by Waterluvian »

I know there's a lot of anecdotal evidence and some fairly clear explanations for why `us-east-1` can be less reliable. But are there any empirical studies that demonstrate this? Like if I wanted to back up this assumption/claim with data, is there a good link for that, showing that us-east-1 is down a lot more often?

Re: AWS multiple services outage in us-east-1

#696

Post by parliament32 »

ljdtt wrote: post #693
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

Yes, although it's military in origin.

Re: AWS multiple services outage in us-east-1

#697

Post by vishnugupta »

ljdtt wrote: post #693
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

It used to be quite common but has fallen out of usage.

Re: AWS multiple services outage in us-east-1

#698

Post by strictnein »

ljdtt wrote: post #693
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

FUBAR: Fucked Up Beyond All Recognition

Somewhat common. Comes from the US military in WW2.

Re: AWS multiple services outage in us-east-1

#700

Post by sorentwo »

ljdtt wrote: post #693
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

first time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)

It is an old US military term that means “F*ked Up Beyond All Recognition”
Post reply on HN