Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

111–120 of 214 posts

Re: AWS North Virginia data center outage – resolved

#111

Post by 999900000999 »

jethro_tell wrote: post #108
999900000999 wrote: post #88

Earlier quoted context omitted.

The idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack. It's basically a wash for almost all organizations for twice the cost and effort.

Ok...

But where does the load balancer actually run. Does load balancer main run on AWS, and load balancer backup on Oracle?

Re: AWS North Virginia data center outage – resolved

#112

Post by adamg203 »

fastest963 wrote: post #64

Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?

spent the evening looking at SLI graphs waiting for the region to blow up but it never did. only a few envs across many had some degraded EBS vols in the single AZ. it was absolutely a single az (use-az4).

Re: AWS North Virginia data center outage – resolved

#113

Post by dlenski »

jethro_tell wrote: post #107
sidewndr46 wrote: post #103

Earlier quoted context omitted.

Isn't this kind of circular dependency what lead to extended downtime a while back?

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

Yes, I concur.

Sometimes the circular dependencies get almost cartoonishly silly.

Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on an S3 bucket."

I made that example up, but only barely.

Re: AWS North Virginia data center outage – resolved

#114

Post by grogenaut »

sidewndr46 wrote: post #103
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Isn't this kind of circular dependency what lead to extended downtime a while back?

when you have a circular dependency, one strategy employed, is to have it be circular but interruptible for 18 or so hours. Call it an oh shit bar.

I'm glad I never had to get that deep into the failure chain.

Re: AWS North Virginia data center outage – resolved

#115

Post by grogenaut »

Eridrus wrote: post #98
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

none of my stuff is in us-east-1. I chose that specifically 15 years ago. Been a great decision.

Re: AWS North Virginia data center outage – resolved

#116

Post by MichaelZuo »

dlenski wrote: post #113
jethro_tell wrote: post #107

Earlier quoted context omitted.

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

Why would such a critical server even be accessible with only one set of keys?

I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

Re: AWS North Virginia data center outage – resolved

#117

Post by grogenaut »

jethro_tell wrote: post #108
999900000999 wrote: post #88

Earlier quoted context omitted.

The idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack. It's basically a wash for almost all organizations for twice the cost and effort.

also these things don't go down THAT often... well aws, not some others. More uptime that you probably had before. even the stock market takes a few days off every decade. Just ask W.

Re: AWS North Virginia data center outage – resolved

#118

Post by electroly »

linsomniac wrote: post #109
Eridrus wrote: post #98

Earlier quoted context omitted.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

>And honestly, everybody else's stuff is in use-1 Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.

Not OP, but I do single-region us-east-1 for a few reasons:

1. The severity and frequency of us-east-1 outages are vastly overstated. It's fine. These us-east-1 outages almost never affect us. This one didn't; not even our instances in the affected AZ. Only that recent IAM outage affected us a little bit, and it affected every other region, too, since IAM's control plane is centrally hosted in us-east-1. Everybody's uptime depends on us-east-1.

2. We're physically close to us-east-1 and have Direct Connect. We're 1 millisecond away from us-east-1. It would be silly to connect to us-east-1 and then take a latency hit and pay cross-region data transfer cost on all traffic to hop over to another region. That would only make sense if we were in both regions, and that is not worth the cost given #1. If we only have a single region, it has to be us-east-1.

3. us-east-1 gets new features first. New AWS features are relevant to us with shocking regularity, and we get it as soon as it's announced.

4. OP is right about the safety in numbers. Our service isn't life-or-death; nobody will die if we're down, so it's just a matter of whether they're upset. When there is a us-east-1 outage, it's headline news and I can link the news report to anyone who asks. That genuinely absolves us every time. When we're down, everybody else is down, too.

Re: AWS North Virginia data center outage – resolved

#119

Post by el_benhameen »

fabian2k wrote: post #4

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool? So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

Good listen on similar topics here: https://signalsandthreads.com/the-thermodynamics-of-trading/

Re: AWS North Virginia data center outage – resolved

#120

Post by wgjordan »

dlenski wrote: post #113
jethro_tell wrote: post #107

Earlier quoted context omitted.

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

A real example, from Facebook's 2021 outage [1]:

> Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.

There was one (later denied) report that a 'guy with an angle grinder' was involved in gaining access to the server cage.

[1] https://news.ycombinator.com/item?id=28762611

Post reply on HN