Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

91–100 of 214 posts

Re: AWS North Virginia data center outage – resolved

#91

Post by dylan604 »

PunchyHamster wrote: post #27
arjie wrote: post #21

Earlier quoted context omitted.

Amusingly I've been part of two critical downtime heating incidents at two different datacenters: one was when Hosting.com's SOMA datacenter got so hot that they were using hoses on the roof to cool it down; and the second one was when Alibaba's Chai Wan datacenter got so hot everything running there went down, including the control plane. So I imagine the proximity to the ocean does not yield any additional advantag…

yeah but capacity is easier/cheaper to build/overbuild if you can access cold-ish water at all times

Didn't really help Fukushima though. In fact, the ocean came to it. They didn't have to go get it.

Re: AWS North Virginia data center outage – resolved

#92

Post by b40d-48b2-979e »

afro88 wrote: post #84
999900000999 wrote: post #75

Earlier quoted context omitted.

Too many people are using it. In fantasy magic dream land loads are distributed evenly across different cloud providers. A single point of failure doesn't exist. It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS. Healthcare in the US is affordable. All types of magical stuff exist here. But no. It's another day…

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

I was surprised recently when setting up cloudfront with aws certs that it forced me to use us-east-1 to provision the certs.

Re: AWS North Virginia data center outage – resolved

#94

Post by merek »

fastest963 wrote: post #64

Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?

I can't find an official source, but I suspect the blast radius isn't limited to the AZ.

I have systems running in us-east-1, and over the course of the incident, I noticed unexplainable intermittent connectivity issues that I've never seen before, even outside of az4.

Re: AWS North Virginia data center outage – resolved

#95

Post by kbbgl87 »

afro88 wrote: post #84
999900000999 wrote: post #75

Earlier quoted context omitted.

Too many people are using it. In fantasy magic dream land loads are distributed evenly across different cloud providers. A single point of failure doesn't exist. It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS. Healthcare in the US is affordable. All types of magical stuff exist here. But no. It's another day…

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

STS is only on us-east-1 I believe

Re: AWS North Virginia data center outage – resolved

#96

Post by jedberg »

adriand wrote: post #77
stingraycharles wrote: post #71

Earlier quoted context omitted.

It really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS. I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware. It’s also the most complex region for A…

What kind of reputation does ca-central-1 have? I’ve been using it and it seems quietly excellent. Knock on wood.

Most of the other regions are fairly stable. Ohio (us-east-2) is a great choice if you're just starting out. Not sure about ca-central-1, but I've never heard anything bad about it.

Re: AWS North Virginia data center outage – resolved

#97

Post by Andys »

fabian2k wrote: post #4

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool? So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

I worked in a DC that had multiple redundant chillers on the roof, and multiple redundant coolers on each floor, but the whole building's cooling failed at once when the water lines failed somehow.

They didn't say how, but apparently the pipes between each floor and the roof were not redundant. It took almost 24 hours to fix.

Re: AWS North Virginia data center outage – resolved

#98

Post by Eridrus »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected.

And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

Re: AWS North Virginia data center outage – resolved

#99

Post by jedberg »

sitzkrieg wrote: post #87

using aws since s3 came out and i’ve yet to see any major company do multi az failover in any capacity whatsoever. default region ftw

We were doing multi-AZ and multi-region failover at Netflix all the way back in 2011:

https://netflixtechblog.com/the-netflix-simian-army-16e57fba...

Re: AWS North Virginia data center outage – resolved

#100

Post by tardedmeme »

dylan604 wrote: post #89
tardedmeme wrote: post #63

Earlier quoted context omitted.

You can't have a duty cycle above 100%. It's impossible.

Not according to POTUS math. You can have 200%, 500%, 600%, 1200%. You just have to say it enough and people will question if they really might not understand percentages enough, and just go with it.

ok but cooling systems don't run on POTUS math though
Post reply on HN