Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

21–30 of 214 posts

Re: AWS North Virginia data center outage – resolved

#21

Post by arjie »

Havoc wrote: post #5

Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat

Amusingly I've been part of two critical downtime heating incidents at two different datacenters: one was when Hosting.com's SOMA datacenter got so hot that they were using hoses on the roof to cool it down; and the second one was when Alibaba's Chai Wan datacenter got so hot everything running there went down, including the control plane. So I imagine the proximity to the ocean does not yield any additional advantage in terms of emergency heat sinking. You have x capacity to pump heat out and it doesn't matter if you're next to the sea or in the middle of Nebraska because your entire system needs to be built to be rated for some performance.

Re: AWS North Virginia data center outage – resolved

#24

Post by PunchyHamster »

AdamJacobMuller wrote: post #14
fabian2k wrote: post #4

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool? So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

The cooling units dont fail just because they get to 100% duty cycle. That's pretty much "normal operation", you just get... higher efficiency coz the cooling side is warmer

Re: AWS North Virginia data center outage – resolved

#25

Post by cperciva »

fabian2k wrote: post #16
AdamJacobMuller wrote: post #14

Earlier quoted context omitted.

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

I'd expect someone like AWS to just throttle machines before overloading their cooling. Because they probably can do that, while e.g. a data center that just rents the space can't really throttle their customers nicely.

Reducing clock speeds, even if they could do that -- and I'm not sure they can, given how Nitro is designed -- would be problematic since a lot of customer workloads assume homogeneous nodes.

But they did load-shed. Perhaps not soon enough, but the reason this is publicly known is because they reduced the amount of heat being produced.

Re: AWS North Virginia data center outage – resolved

#26

Post by AdamJacobMuller »

foota wrote: post #19
AdamJacobMuller wrote: post #14

Earlier quoted context omitted.

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

Shouldn't there be a feedback system here preventing the scheduling of loads when cooling is degraded?

With hyperscalers for sure.

But this is the physical world, shit happens.

The algorithm didn't know that fuse was lose and fine at 50% duty cycle but was high resistance and going to blow at 100%.

Re: AWS North Virginia data center outage – resolved

#27

Post by PunchyHamster »

arjie wrote: post #21
Havoc wrote: post #5

Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat

Amusingly I've been part of two critical downtime heating incidents at two different datacenters: one was when Hosting.com's SOMA datacenter got so hot that they were using hoses on the roof to cool it down; and the second one was when Alibaba's Chai Wan datacenter got so hot everything running there went down, including the control plane. So I imagine the proximity to the ocean does not yield any additional advantag…

yeah but capacity is easier/cheaper to build/overbuild if you can access cold-ish water at all times

Re: AWS North Virginia data center outage – resolved

#28

Post by AdamJacobMuller »

fabian2k wrote: post #16
AdamJacobMuller wrote: post #14

Earlier quoted context omitted.

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

I'd expect someone like AWS to just throttle machines before overloading their cooling. Because they probably can do that, while e.g. a data center that just rents the space can't really throttle their customers nicely.

Its harder and harder to throttle machines with hardware segmentation capabilities effectively passing through hardware components "intact"

A decade ago it was trivial to just tell the hypervisor to reduce the cpu fraction of all VMs by half and leave half unallocated. Now, it's much more complicated and definitely would be user visible.

Re: AWS North Virginia data center outage – resolved

#29

Post by AdamJacobMuller »

PunchyHamster wrote: post #24
AdamJacobMuller wrote: post #14

Earlier quoted context omitted.

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

The cooling units dont fail just because they get to 100% duty cycle. That's pretty much "normal operation", you just get... higher efficiency coz the cooling side is warmer

Of course not. They fail above 100%.

Some fail below 100% too.

Re: AWS North Virginia data center outage – resolved

#30

Post by tempaccount5050 »

Havoc wrote: post #5

Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat

Lots of proposals to build them near Lake Michigan recently but the residents of Wisconsin only want auto parts stores and paper mills. They've been completely demonized. Cities and counties are passing no data center laws even though it's the perfect place for it.
Post reply on HN