Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Operational issue – Multiple services (UAE)

health.aws.amazon.com

Post reply on HN
Print view View original

41–50 of 125 posts

Re: Operational issue – Multiple services (UAE)

#41

Post by swiftcoder »

p-o wrote: post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

The more interesting question, is how many datacenters are just plonked next to a high-value military target?

For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

Re: Operational issue – Multiple services (UAE)

#42

Post by jcgrillo »

arjie wrote: post #38
eptcyka wrote: post #16

> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. God forbid we'd ever say that it was struck by a missile or a munition in an act of war.

I actually like the way they said it. I don't know if it's a different cultural tradition, but the cool steely-eyed fact-based conversation always really felt so much more inspiring: Conrad: I got three fuel cell lights, an AC bus light, a fuel cell disconnect, AC bus overload 1 and 2, Main Bus A and B out. Aaron: Flight, EECOM. Try SCE to Aux. Modern culture in the movies and whatnot is that someone should be yellin…

High signal/noise ratio is extra important when things are going badly

Re: Operational issue – Multiple services (UAE)

#43

Post by SoftTalker »

Imustaskforhelp wrote: post #23

Has this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was cause…

Data centers are usually built to withstand local natural risks e.g. weather. All bets, SLAs, and insurance are usually off when it comes to acts of war.

Re: Operational issue – Multiple services (UAE)

#44

Post by roxolotl »

p-o wrote: post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

This is the data center version of https://xkcd.com/538/. Realistically if there is a hot war what you’re saying seems accurate.

Re: Operational issue – Multiple services (UAE)

#45

Post by roncesvalles »

p-o wrote: post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

Exactly. 2 is only sufficient for HA against random failures. It's not enough for HA against a determined adversary willing to use targeted force.

Re: Operational issue – Multiple services (UAE)

#46

Post by throw475787 »

swiftcoder wrote: post #41
p-o wrote: post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

The more interesting question, is how many datacenters are just plonked next to a high-value military target? For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

Is there acrually some meaningful physical separation between military and civilian server deployments?

We seem to be really bad at separating those two. For example Starlink is basically military infrastructure now, used to guide bombs.

Re: Operational issue – Multiple services (UAE)

#47

Post by tokyobreakfast »

Imustaskforhelp wrote: post #34
bgentry wrote: post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

> this would require human intervention

that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month.

same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

Re: Operational issue – Multiple services (UAE)

#48

Post by jiggawatts »

bgentry wrote: post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…

Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.

A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.

“Oh, that’s in case there’s a fire!”

“What does it do?”

“It cuts… the power… uhh… for the safety of the fire department.”

“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”

“… yes.”

“Not 100% then, is it?”

Re: Operational issue – Multiple services (UAE)

#49

Post by cherryteastain »

swiftcoder wrote: post #41
p-o wrote: post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

The more interesting question, is how many datacenters are just plonked next to a high-value military target? For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

A datacenter IS a high value military target.

Re: Operational issue – Multiple services (UAE)

#50

Post by flymasterv »

tokyobreakfast wrote: post #47
Imustaskforhelp wrote: post #34

Earlier quoted context omitted.

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.
Post reply on HN