Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

191–200 of 214 posts

Re: AWS North Virginia data center outage – resolved

#191

Post by bink »

nilamo wrote: post #181
Eridrus wrote: post #98

Earlier quoted context omitted.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

> And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol. Is it not a selling point to be able to say "we're still up while out competitors are down"?

It's worse when your region has issues and your customer's infrastructure is fine.

Re: AWS North Virginia data center outage – resolved

#192

Post by bink »

Leonard_of_Q wrote: post #39
tempaccount5050 wrote: post #30

Earlier quoted context omitted.

Lots of proposals to build them near Lake Michigan recently but the residents of Wisconsin only want auto parts stores and paper mills. They've been completely demonized. Cities and counties are passing no data center laws even though it's the perfect place for it.

Paper mills need a lot of heat energy to run the processes. Data centres produce a lot of heat. Sounds like a good combination? Cold water -> data centre cooling loop - > warm water -> paper mill with heat pumps to transform low-grade heat into the required temperatures -> profit

I can't believe any town would vote for a paper mill. It smells like a paper mill.

Re: AWS North Virginia data center outage – resolved

#193

Post by skywhopper »

Eridrus wrote: post #98
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

It wasn’t even all of a single AZ. None of my resources in use1-az4 had any issues. The most annoying thing was the 20 notifications we got saying “it’s not all fixed yet” every hour.

Re: AWS North Virginia data center outage – resolved

#194

Post by skywhopper »

nilamo wrote: post #181
Eridrus wrote: post #98

Earlier quoted context omitted.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

> And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol. Is it not a selling point to be able to say "we're still up while out competitors are down"?

If you’re the one that’s down while no one else is, suddenly it becomes your fault.

Re: AWS North Virginia data center outage – resolved

#195

Post by [deleted] »

bigfatkitten wrote: post #123
MichaelZuo wrote: post #116

Earlier quoted context omitted.

Why would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

Other than for certain nuclear missile launches[1], that only happens in the movies. [1] https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact...

[deleted]

Re: AWS North Virginia data center outage – resolved

#196

Post by dlenski »

myroon5 wrote: post #159
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

> outside of China [Nitpick] There are a few more AWS partitions like GovCloud: https://jasonbutz.info/2023/07/aws-partitions/

Yes, I'm certainly aware of the other partitions. That's why I said all the public cloud regions outside China.

Yeah, "govcloud" is technically available to the public, although there are other partitions reserved for government use that are not, and the naming is a big hairy mess. Many service teams don't have any US-citizens-in-the-USA working for them, and they cannot in any way adequately support these regions.

My on-call experience improved significantly when I moved from the US to Canada, and I got taken off the (extremely thin!) list of engineers eligible to ssh into RDS instances in Govcloud. There were so few USA-citizen-in-USA engineers that I had been getting tickets for services and instances in Govcloud about which I had only the very thinnest knowledge… and then I was limited in my ability to consult with others who were actually experts. The customers in Govcloud paid a premium to be there, I got paged for a bunch of tickets which I was ill-prepared to handle, and it was generally a bad experience for everyone.

Working with the airgapped secret/top-secret partitions was even worse. You would get paged incessantly and then someone who was cleared for access but knew almost nothing about the service in question would have to go to a SCIF in the DC area, and you would exchange screenshots and text instructions with a turnaround time of hours or days.

Re: AWS North Virginia data center outage – resolved

#197

Post by dlenski »

grogenaut wrote: post #114
sidewndr46 wrote: post #103

Earlier quoted context omitted.

Isn't this kind of circular dependency what lead to extended downtime a while back?

when you have a circular dependency, one strategy employed, is to have it be circular but interruptible for 18 or so hours. Call it an oh shit bar. I'm glad I never had to get that deep into the failure chain.

> Call it an oh shit bar.

Amazon's equivalent of this (sort of) was the "andon cord."

Not only was the physical metaphor that led to this name never properly explained (it's basically an "emergency stop" string in a Toyota factory), but the actual use of this mechanism was so heavily discouraged that I never saw it used in 4+ years at Amazon. except once very performatively by a VP who had already been paged awake at 2am or something like that.

In my experience, a lot of the AWS engineers live in continuous fear of screwing up by using the huge array of extremely powerful, dangerous, poorly-explained, and ever-changing tools that they have access to.

Re: AWS North Virginia data center outage – resolved

#198

Post by dlenski »

avereveard wrote: post #129
dlenski wrote: post #102

Earlier quoted context omitted.

Yep. All of the identity and access management services for the non-China public cloud are in us-east-1. https://news.ycombinator.com/item?id=48071472

All the control plane . Data plane is distributed and roles using iam to access resources can still do so during a control plane outage.

Yes, you're right, but in my experience the boundary between the data plane and the control plane is not always clear, and especially unclear on these foundational and basic services.

There were enough "surprisingly control-plane" IAM operations in the AWS services that I dealt with, so we had to exercise extreme caution during outages.

Re: AWS North Virginia data center outage – resolved

#199

Post by dlenski »

jmsgwd wrote: post #164
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

> The idea that AWS's services are fully regionalized or isolated has always been a myth. This is highly misleading. It's true that there's a handful of global AWS services - but only their control planes operate from a single region (e.g. us-east-1). Their data planes are regionally isolated or globally distributed.[1] The only time you'd normally use a service control plane is to deploy changes, e.g. when you creat…

You're right of course to distinguish the control plane and data plane, and it sounds like you know more about this than I do for IAM.

I disagree, though, that my post was "highly misleading" despite this omission.

As a practical matter, some services fail to achieve the "static stability" you describe, in terms of not depending on other services’ control planes.

And also, many on-calls ops and firefighting tasks (to say nothing of canaries and other automated tests) depend on other services’ control planes.

And above all, many AWS engineers (myself very much included even after years there) don't have a clear understanding of the boundaries of other services’ control planes. https://news.ycombinator.com/item?id=48078254

> > During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones.

> This is just plain wrong! The IAM Security Token Service (STS), which grants IAM tokens, is a data plane-only service and runs independently in each region.

I didn't mention STS in the service to which you're responding. The service that I worked on the most, RDS, required ssh'ing into live instances to solve basically all non-trivial problems (I'd guess 80% of the tickets that I saw actually resolved required it). And I have no idea if it how STS was involved in generating the ephemeral Midway-signed ssh keys required for it… but whenever there were us-east-1 IAM outages we'd have big problems opening new sessions, while less-capable web-console-based ops tools with long-lived credentials would keep working.

Re: AWS North Virginia data center outage – resolved

#200

Post by everfrustrated »

dlenski wrote: post #198
avereveard wrote: post #129

Earlier quoted context omitted.

All the control plane . Data plane is distributed and roles using iam to access resources can still do so during a control plane outage.

Yes, you're right, but in my experience the boundary between the data plane and the control plane is not always clear, and especially unclear on these foundational and basic services. There were enough "surprisingly control-plane" IAM operations in the AWS services that I dealt with, so we had to exercise extreme caution during outages.

It's literally documented. Try reading it and educating yourself.
Post reply on HN