61–70 of 254 posts
Wonder if this is why Zoom is down. Wasn't able to connect just now. The connection proxy/sites were giving 504s.
https://www.oracle.com/customers/zoom/
I'm not sure if Zoom has any Critical infra in AWS though.
I'm running Terraform and it appears to be stuck now. What do I do??
Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.
$ dig news.ycombinator.com
;; ANSWER SECTION:
news.ycombinator.com. 1 IN A 50.112.136.166
$ dig -x 50.112.136.166
;; ANSWER SECTION:
166.136.112.50.in-addr.arpa. 300 IN PTR ec2-50-112-136-166.us-west-2.compute.amazonaws.com.
Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!
Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…
Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!
Earlier quoted context omitted.
Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.
Thanks, this comment made it very clear to me that I never want to touch a terraform system.
Whether TF can update the state & release its locks would depend on where those were hosted. If they're in the downed AZ, then ofc. it can't do that, and manual intervention will be required afterwards. I forget if you can make those objects regional when stored in AWS or not. (You can in some other storages.)
… what would you expect to happen here?
Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!
In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?
us-west-2 has had outages as well but it is less common, even rare. I've been pushing companies to make their initial deployments onto us-west-2 for over ten years now. I occasionally get kudos messages in my inbox :)