Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

71–80 of 214 posts

Re: AWS North Virginia data center outage – resolved

#71

Post by stingraycharles »

cherioo wrote: post #60
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

Is it really failing more, or we just don’t hear about failure happening elsewhere? Last i heard azure outage it wasn’t even on HN frontpage

It really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS.

I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware.

It’s also the most complex region for AWS themselves, as it’s the “control pad” for many of their global services.

Re: AWS North Virginia data center outage – resolved

#72

Post by AdamJacobMuller »

cperciva wrote: post #25
fabian2k wrote: post #16

Earlier quoted context omitted.

I'd expect someone like AWS to just throttle machines before overloading their cooling. Because they probably can do that, while e.g. a data center that just rents the space can't really throttle their customers nicely.

Reducing clock speeds, even if they could do that -- and I'm not sure they can, given how Nitro is designed -- would be problematic since a lot of customer workloads assume homogeneous nodes. But they did load-shed. Perhaps not soon enough, but the reason this is publicly known is because they reduced the amount of heat being produced.

> But they did load-shed

Right, exactly, I highly doubt the facility went into any kind of actual uncontrolled thermal rise. This is news because they had to take such drastic actions. I'm sure its common that they force spot prices up (probably way up) to compensate for reduced capacity due to events, I'm sure they even sometimes fake no capacity for similar reasons. No capacity means "I don't want to turn on your node" not merely "I don't have any more physical servers I could turn up for you".

This is news because they powered off some non-preemptible customer loads, which actually makes me wonder if you saw that chain of events occur here.

spot prices rise -> new instance availability goes to 0 -> preemptible instances go dark -> normal instances go dark.

Re: AWS North Virginia data center outage – resolved

#73

Post by 8organicbits »

qaq wrote: post #57
keeganpoppen wrote: post #42

Earlier quoted context omitted.

anecdotally (well, more "second-hand-ly i heard that..." it sounds like there were some carry-on effects on us-east-2 as a result of people migrating over from us-east-1, so, yeah... kinda hilarious how the multiple region / AZ thing is just so plainly a façade, but yet we all seem to just collectively believe in it as an article of faith in the Cloud Religion... or whatever...

It's no magic given the size of us-east-1 there is no spare capacity to absorb all the workloads

One of the SRE tricks is to reserve your capacity so when the cloud runs out of capacity you're still covered. It's expensive, but you don't want to get stuck without a server when the on-demand dries up.

Re: AWS North Virginia data center outage – resolved

#75

Post by 999900000999 »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

Too many people are using it.

In fantasy magic dream land loads are distributed evenly across different cloud providers.

A single point of failure doesn't exist.

It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS.

Healthcare in the US is affordable.

All types of magical stuff exist here.

But no. It's another day. AWS US-East 1 can take town most of the internet.

Re: AWS North Virginia data center outage – resolved

#76

Post by ckozlowski »

mandevil wrote: post #9
Havoc wrote: post #5

Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too Two loop cycle with heat exchanger to get rid of the heat

So Ashburn VA is a datacenter hub because the very first non-government Internet Exchange Point (IXP) anywhere in the world was there ( https://en.wikipedia.org/wiki/MAE-East ). Back in the 1990's something like half of all internet traffic all over the world hit MAE-East. That in turn made AWS put their first region there (us-east-1 preceded eu-west-1 by 2 years and us-west-1 by 3 years). Then because there were lot…

Well said. I'll also add, that with these networks, the sooner you can get traffic off your network the better. There's strong incentive to have your datacenter near these peering points. And since MAE-East was the first, it's been the largest as it's been snowballing the oldest. AOL's HQ was here, Equinix built their peering point soon after MAE-East, etc.

There's a great read about the whole area here: https://www.amazon.com/Internet-Alley-Technology-1945-2005-I...

As for AWS, I often see it repeated that the DCs are the oldest and therefor in disrepair. That's not true; many of the first ones have since been replaced. But there are services that are located here and only here.

But I'll also add, a lot of customers default to using US-East-1 without considering others, and too many deploy in only one AZ. Part of this is AWS's fault as their new services often launch in US-East-1 and West-2 first, so customers go to East-1 to get the new features first.

Speaking as one who was with AWS for 10 years as a TAM and Well-Architected contributor, I saw a lot of customers who didn't design with too much resiliency in mind, and so they get adversely affected when east-1 has an issue (either regional or AZ). The other regions have their fair bit of issues as well. It's not so much that east-1 necessarily fails more than the others, it's that it has so many AZs and so many workloads that people notice it more.

Re: AWS North Virginia data center outage – resolved

#77

Post by adriand »

stingraycharles wrote: post #71
cherioo wrote: post #60

Earlier quoted context omitted.

Is it really failing more, or we just don’t hear about failure happening elsewhere? Last i heard azure outage it wasn’t even on HN frontpage

It really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS. I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware. It’s also the most complex region for A…

What kind of reputation does ca-central-1 have? I’ve been using it and it seems quietly excellent. Knock on wood.

Re: AWS North Virginia data center outage – resolved

#78

Post by morgoths_bane »

shimman wrote: post #43
aurareturn wrote: post #37

These things are dangerous. Someone who can take AWS down such as an employee can place a bet. These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.

It's a good thing big tech hires for ethical engineers and not ones that only care about money or social status.

Thankfully their leadership is leading the way in ethics since inception, so I am confident that no such shenanigans will ever take place. I may even bet on this.

Re: AWS North Virginia data center outage – resolved

#79

Post by unethical_ban »

tailscaler2026 wrote: post #17

us-east-1 is down? shocking! stop putting SPOF services there. this location has had frequent issues for the past 15 years.

This is correct... unless there is a specific requirement to be in that location for some kind of IXP or ultra low latency, I can't imagine putting mission-critical things in only that region.

Re: AWS North Virginia data center outage – resolved

#80

Post by __turbobrew__ »

tom1337 wrote: post #58
corvad wrote: post #51

It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.

Isn't east one the "core" datacenter and also the oldest? I'd imagine it has more load than the other regions and also has more tech debt and architectural / engineering debt because they had less experience when they built it. Also iirc some services rely on east-1 as a single point of failure for configuration (like IAM or some S3 stuff?)

What I have seen at other companies is that the older datacenters have suboptimal designs which are impossible to fix after the fact.
Post reply on HN