Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

101–110 of 214 posts

Re: AWS North Virginia data center outage – resolved

#101

Post by dlenski »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth.

All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions.

And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM.

During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones. When I worked there, I remember at least one case where my team's on-calls were advised not to close ssh sessions or AWS console browser tabs, for fear that we'd be locked out until the outage was over.

Re: AWS North Virginia data center outage – resolved

#102

Post by dlenski »

kbbgl87 wrote: post #95
afro88 wrote: post #84

Earlier quoted context omitted.

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

STS is only on us-east-1 I believe

Yep. All of the identity and access management services for the non-China public cloud are in us-east-1. https://news.ycombinator.com/item?id=48071472

Re: AWS North Virginia data center outage – resolved

#103

Post by sidewndr46 »

dlenski wrote: post #101
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Isn't this kind of circular dependency what lead to extended downtime a while back?

Re: AWS North Virginia data center outage – resolved

#104

Post by dlenski »

adriand wrote: post #77
stingraycharles wrote: post #71

Earlier quoted context omitted.

It really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS. I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware. It’s also the most complex region for A…

What kind of reputation does ca-central-1 have? I’ve been using it and it seems quietly excellent. Knock on wood.

It wasn't heavily utilized when I worked at AWS, until 2024.

If your customers are clusterrd in Toronto and Montreal, it probably makes a lot of sense to use ca-central-1. If you've got a lot of customers in Western Canada, us-west-2 is gonna have better network latency.

Other than a couple regions that had problems with their local network infrastructure (sa-east-1 was like that), there's little or nothing to differentiate the regions in terms of physical infrastructure and architecture.

Re: AWS North Virginia data center outage – resolved

#105

Post by dylan604 »

tardedmeme wrote: post #100
dylan604 wrote: post #89

Earlier quoted context omitted.

Not according to POTUS math. You can have 200%, 500%, 600%, 1200%. You just have to say it enough and people will question if they really might not understand percentages enough, and just go with it.

ok but cooling systems don't run on POTUS math though

Nor does the rest of the world

Re: AWS North Virginia data center outage – resolved

#106

Post by dtjohnnymonkey »

AdamJacobMuller wrote: post #14
fabian2k wrote: post #4

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool? So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

I could totally get into “Ops Thriller” genre of novels like this.

Re: AWS North Virginia data center outage – resolved

#107

Post by jethro_tell »

sidewndr46 wrote: post #103
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Isn't this kind of circular dependency what lead to extended downtime a while back?

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

Re: AWS North Virginia data center outage – resolved

#108

Post by jethro_tell »

999900000999 wrote: post #88
afro88 wrote: post #84

Earlier quoted context omitted.

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

The idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack.

It's basically a wash for almost all organizations for twice the cost and effort.

Re: AWS North Virginia data center outage – resolved

#109

Post by linsomniac »

Eridrus wrote: post #98
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

>And honestly, everybody else's stuff is in use-1

Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.

Re: AWS North Virginia data center outage – resolved

#110

Post by christina97 »

linsomniac wrote: post #109
Eridrus wrote: post #98

Earlier quoted context omitted.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

>And honestly, everybody else's stuff is in use-1 Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.

But it’s okay to be down when the whole internet is down.
Post reply on HN