Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 254 posts

Re: AWS us-east-2 outage

#71

Post by throwawaymaths »

alfalfasprout wrote: post #66
packetslave wrote: post #45

Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

Insert clip of O'Brien explaining to cardassians why there are backups for backups

Re: AWS us-east-2 outage

#72

Post by teknopaul »

arecurrence wrote: post #70
acwan93 wrote: post #65

Earlier quoted context omitted.

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

Classically, us-east-1 received most of the hate given its immense size (it used to be several times larger than any other) and status as the first large aws data center. It also seemed to launch new aws features first but that may have been my imagination. If true, I'm sure always running the latest builds was not great for stability. us-west-2 has had outages as well but it is less common, even rare. I've been push…

93.99999

Re: AWS us-east-2 outage

#73

Post by jmartens »

alfalfasprout wrote: post #66
packetslave wrote: post #45

Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

Ya, am I surprised by this too. Like, you have one job, keep the power on.

Re: AWS us-east-2 outage

#74

Post by packetslave »

alfalfasprout wrote: post #66
packetslave wrote: post #45

Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone. 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the major…

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

It depends entirely on how AWS architected their power redundancy. Given that the outage affected a portion of one DC in one AZ, we can make some assumptions, but the truth is we just don't know.

It could be that their shared-fate scope is an entire data hall, or a set of rows, or even an entire building given that an AZ is made up of multiple datacenters. I don't know that AWS has ever published any kind of sub-AZ guarantees around reliability.

Datacenter power has all kinds of interesting failure modes. I've seen outages caused by a cat climbing into a substation, rats building a nest in a generator, fire-fighting in another part of the building causing flooding in the high-voltage switching room, etc.

Re: AWS us-east-2 outage

#75

Post by EddySchauHai »

Syonyk wrote: post #40
corobo wrote: post #9

Ahh Always check HN before trying to diagnose weird issues that shouldn't be connected

And the reason that works is because HN is mostly hosted on its own stuff, without weird dependencies on anything beyond "the servers being up" and "TCP mostly working."

I believe it's on AWS after its two servers broke at the same time the other day.

Re: AWS us-east-2 outage

#77

Post by latchkey »

packetslave wrote: post #74
alfalfasprout wrote: post #66

Earlier quoted context omitted.

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

It depends entirely on how AWS architected their power redundancy. Given that the outage affected a portion of one DC in one AZ, we can make some assumptions, but the truth is we just don't know. It could be that their shared-fate scope is an entire data hall, or a set of rows, or even an entire building given that an AZ is made up of multiple datacenters. I don't know that AWS has ever published any kind of sub-AZ g…

Our best was a bird landing on a transformer up on a pole. Installed a fake Eagle after that.

Re: AWS us-east-2 outage

#78

Post by linsomniac »

I understand that us-east is AWS's oldest and biggest facility, but Amazon seems to have more money than Croesus, why aren't they fixing/rebuilding/replacing us-east with something more modern?

Re: AWS us-east-2 outage

#79

Post by jedberg »

acwan93 wrote: post #65
jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

It's never been a default datacenter. For a long time the default when you first logged into the console was us-east-1 so a lot of companies set up there (that's where all of reddit was run for a long time and Netflix too). At some point they switched the default to us-east-2.

So anyone who is in us-west-2 is there intentionally, which makes me assume there is a smaller footprint there (but I have no idea).

Post reply on HN