Looks like a lot of services are impacted including Cloudflare, Ping, Zoom and Datadog.
Cloudflare uses AWS? For what?
AWS us-east-2 outage
81–90 of 254 posts
Re: AWS us-east-2 outage
#82Had Rackspace login issues earlier today. Hmmmmmm ...
Re: AWS us-east-2 outage
#83I understand that us-east is AWS's oldest and biggest facility, but Amazon seems to have more money than Croesus, why aren't they fixing/rebuilding/replacing us-east with something more modern?
us-east is a geographic distinction within which there are multiple regions. us-east-1 and us-east-2 are not the same. This outage occurred in us-east-2. Within an AWS region there are multiple data centers. They call their data centers availability zones. The availability zone AZ1 was the one impacted, and within that availability zone, most likely only a subset of servers.
us-east-1 is the region you're thinking of that has issues. Mostly due to being the largest region (I think?) and like you mentioned, the oldest.
Re: AWS us-east-2 outage
#84Earlier quoted context omitted.
Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.
Thanks, this comment made it very clear to me that I never want to touch a terraform system.
I think any system is susceptible to problems like this if the underlying hardware becomes unavailable. Using dynambodb to obtain locks on s3 is a pretty common pattern in AWS development. This has more to do with AWS than Terraform.
Re: AWS us-east-2 outage
#85I understand that us-east is AWS's oldest and biggest facility, but Amazon seems to have more money than Croesus, why aren't they fixing/rebuilding/replacing us-east with something more modern?
us-east is a geographic distinction within which there are multiple regions. us-east-1 and us-east-2 are not the same. This outage occurred in us-east-2. Within an AWS region there are multiple data centers. They call their data centers availability zones. The availability zone AZ1 was the one impacted, and within that availability zone, most likely only a subset of servers. us-east-1 is the region you're thinking of…
> us-east-1 is the region you're thinking of that has issues. Mostly due to being the largest region (I think?) and like you mentioned, the oldest.
Also, because shared and global AWS resource are (or at least often behave as if they are) intimately tied to us-east-1.
Re: AWS us-east-2 outage
#86Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…
I thought that AWS availability zones were intentionally not canonically named to prevent everyone from adding stuff to AZ “A”. So my us-east-1 zone “A” might be your “B”.
But that system breaks down here when you need to know whether you are in an affected zone. Is there a way to map an account’s AZ name to the canonical one which apparently exists?
Re: AWS us-east-2 outage
#87We have always been at war with us-east-2.
Re: AWS us-east-2 outage
#88Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…
I thought that AWS availability zones were intentionally not canonically named to prevent everyone from adding stuff to AZ “A”. So my us-east-1 zone “A” might be your “B”. But that system breaks down here when you need to know whether you are in an affected zone. Is there a way to map an account’s AZ name to the canonical one which apparently exists?
Re: AWS us-east-2 outage
#89Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!
In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?
Rather the opposite - us-west-2 is big but not the biggest region, or the smallest, or the oldest or newest, it's not partitioned off like the China or GovCloud regions. Because us-west-2 is fairly typical it tends to be one of the last regions to get software updates, after they've been tested in prod elsewhere
Re: AWS us-east-2 outage
#90Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…
I thought that AWS availability zones were intentionally not canonically named to prevent everyone from adding stuff to AZ “A”. So my us-east-1 zone “A” might be your “B”. But that system breaks down here when you need to know whether you are in an affected zone. Is there a way to map an account’s AZ name to the canonical one which apparently exists?
They gave up on that, now there's an extra "zone ID" you can read that maps to an absolute address. They used to be extremely cagey about giving out those mappings for your account.
Examples about how these relate: https://stackoverflow.com/questions/63283340/aws-map-between...
I'm also pretty sure that GCP's identifiers are absolute (and this time, throughout) as well, since their documentation (which renders the same in incognito mode or whatever) makes reference to what zones have what microarchitectures and instance types.