AWS us-east-2 outage • Hacker News Board

Re: AWS us-east-2 outage

#41

Post by dang » Thu, Jul 28, 2022, 5:30 PM UTC

corobo wrote: post #9

Ahh Always check HN before trying to diagnose weird issues that shouldn't be connected

Living a bit more dangerously at the moment as HN is still running temporarily on AWS. (I'd link to the threads about this from a few weeks ago but am on my phone ATM.)

Re: AWS us-east-2 outage

#42

Post by bkruse » Thu, Jul 28, 2022, 5:30 PM UTC

Lots of issues in us-east-2 for instances for us but also other regions when connecting to RDS

Re: AWS us-east-2 outage

#43

Post by monocasa » Thu, Jul 28, 2022, 5:32 PM UTC

aaur0 wrote: post #20

AWS reported the outage here : https://health.aws.amazon.com/health/status

> Severity

> Informational

lol

Re: AWS us-east-2 outage

#44

Post by jedberg » Thu, Jul 28, 2022, 5:32 PM UTC

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2."

Now I guess we have to move to us-west-2. :)

Update: looks like it's only one zone anyway, so my statement still stands!

Re: AWS us-east-2 outage

#45

Post by packetslave » Thu, Jul 28, 2022, 5:34 PM UTC

Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone.

10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.

Re: AWS us-east-2 outage

#46

Post by fragmede » Thu, Jul 28, 2022, 5:34 PM UTC

CoastalCoder wrote: post #37

svnpenn wrote: post #25

Earlier quoted context omitted.

Looks like Snap, Crackle and Pop are down as well.

> Looks like Snap, Crackle and Pop are down as well. I don't work on cloud stuff, so I'm genuinely unsure if this is a joke.

It's a joke but I only knew that because Snap is/was (as of S1) hosted on GCP and not AWS. Crackle happens to be the name of a video on demand company.

It's a reference to the breakfast cereal of the same name.

Re: AWS us-east-2 outage

#47

Post by rwky » Thu, Jul 28, 2022, 5:36 PM UTC

allenrb wrote: post #38

kaustubhvp wrote: post #8

AWS acknowledged the issue https://health.aws.amazon.com/health/status

[10:25 AM PDT] We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of…

Funny I failed away from the zone and RDS still doesn't work, connections fail.

Edit: 2 minutes after I post this it starts working.

Re: AWS us-east-2 outage

#48

Post by nodesocket » Thu, Jul 28, 2022, 5:37 PM UTC

jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Jinxed for sure. I refuse to deploy resource into us-east-1 unless required by the service.

Re: AWS us-east-2 outage

#49

Post by packetslave » Thu, Jul 28, 2022, 5:37 PM UTC

On the bright side, this is certainly a good test for "exactly how resilient are our AWS-based systems to the loss of a single availability zone?"

Re: AWS us-east-2 outage

#50

Post by danw1979 » Thu, Jul 28, 2022, 5:39 PM UTC

hakube wrote: post #21

I'm running Terraform and it appears to be stuck now. What do I do??

Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.