Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

121–130 of 214 posts

Re: AWS North Virginia data center outage – resolved

#121

Post by dboreham »

999900000999 wrote: post #111
jethro_tell wrote: post #108

Earlier quoted context omitted.

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack. It's basically a wash for almost all organizations for twice the cost and effort.

Ok... But where does the load balancer actually run. Does load balancer main run on AWS, and load balancer backup on Oracle?

Short TTL DNS or BGP anycast.

Re: AWS North Virginia data center outage – resolved

#122

Post by lukeify »

AdamJacobMuller wrote: post #14
fabian2k wrote: post #4

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool? So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

This is almost definitely an issue of equipment failure. Cooling in datacenters is like everything else both over and under provisioned. It's overprovisioned in the sense that the big heat exchange units are N+1 (or in very critical and smaller load facilities 2N/3N). This is done because you need to regularly take these down for maintenance work and they have a relatively high failure rate compared to traditional DC…

This is written beautifully. It's like a much more inconsequential variant of Chernobyl.

Re: AWS North Virginia data center outage – resolved

#123

Post by bigfatkitten »

MichaelZuo wrote: post #116
dlenski wrote: post #113

Earlier quoted context omitted.

Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on a…

Why would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

Other than for certain nuclear missile launches[1], that only happens in the movies.

[1] https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact...

Re: AWS North Virginia data center outage – resolved

#124

Post by erikerikson »

999900000999 wrote: post #88
afro88 wrote: post #84

Earlier quoted context omitted.

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

The idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.

Not really. Your clients can random robin to connection points across providers and move write heads upon connection. If you worry about hard coding you can reduce the surface to a per-context first minimum contact point.

Re: AWS North Virginia data center outage – resolved

#125

Post by bombcar »

AdamJacobMuller wrote: post #69
minimaltom wrote: post #32

Earlier quoted context omitted.

This is a great writeup! thank you!! Reminds when i did noogler training back in the day and one of the talks described a cascading failure at a datacenter, starting with a cat which was too curious near a power conditioner, and briefly conducted

The cat incident at a facility I worked at. Its cold up here in the winter, sadly, the residual heat from even totally passive components like switch gear is enough to warm things up enough to attract them. .001% of 1MW of power is still quite warm. (I have no idea how much switchgear leaks but i know they are warm even in winter outdoors). And, yeah, the rest of the writeup is also an amalgamation of some panic-indu…

https://x.com/visakanv/status/1678745111411212290

Re: AWS North Virginia data center outage – resolved

#126

Post by bombcar »

fastest963 wrote: post #64

Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?

East-1 going down always takes some things from other AZs, because there's always something dependent on East-1.

Re: AWS North Virginia data center outage – resolved

#128

Post by stephenr »

dlenski wrote: post #101
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

> And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM.

When you dogfood your own Rube Goldberg machine.

Re: AWS North Virginia data center outage – resolved

#129

Post by avereveard »

dlenski wrote: post #102
kbbgl87 wrote: post #95

Earlier quoted context omitted.

STS is only on us-east-1 I believe

Yep. All of the identity and access management services for the non-China public cloud are in us-east-1. https://news.ycombinator.com/item?id=48071472

All the control plane. Data plane is distributed and roles using iam to access resources can still do so during a control plane outage.

Re: AWS North Virginia data center outage – resolved

#130

Post by wmf »

aussieguy1234 wrote: post #49

Once known for having super reliable services, I've heard this company is scrambling to re hire some of the engineers they overconfidently "replaced" with AI. When customers pay for cloud services, they expect them to be maintained by competent engineers. edit: Not sure why the downvotes. If you fire the engineers that have been keeping your systems running reliably for years, what do you expect to happen?

It's a cooling equipment failure. Equipment is going to fail.
Post reply on HN