Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Operational issue – Multiple services (UAE)

health.aws.amazon.com

Post reply on HN
Print view View original

51–60 of 125 posts

Re: Operational issue – Multiple services (UAE)

#51

Post by ok_dad »

flymasterv wrote: post #50
tokyobreakfast wrote: post #47

Earlier quoted context omitted.

> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.

What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.

Re: Operational issue – Multiple services (UAE)

#52

Post by thatguy0900 »

tokyobreakfast wrote: post #47
Imustaskforhelp wrote: post #34

Earlier quoted context omitted.

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.

Re: Operational issue – Multiple services (UAE)

#53

Post by crote »

Imustaskforhelp wrote: post #23

Has this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was cause…

> will there be a demand (all be small) for nuclear bunker esque datacenters

Those already exist. See for example Bahnhof's "Pionen - White Mountain" data center in Stockholm, or Cyberfort's "The Bunker" a bit west of London.

Re: Operational issue – Multiple services (UAE)

#54

Post by tokyobreakfast »

thatguy0900 wrote: post #52
tokyobreakfast wrote: post #47

Earlier quoted context omitted.

> this would require human intervention that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month. same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.

on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely

Re: Operational issue – Multiple services (UAE)

#55

Post by AlexErrant »

arjie wrote: post #38
eptcyka wrote: post #16

> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. God forbid we'd ever say that it was struck by a missile or a munition in an act of war.

I actually like the way they said it. I don't know if it's a different cultural tradition, but the cool steely-eyed fact-based conversation always really felt so much more inspiring: Conrad: I got three fuel cell lights, an AC bus light, a fuel cell disconnect, AC bus overload 1 and 2, Main Bus A and B out. Aaron: Flight, EECOM. Try SCE to Aux. Modern culture in the movies and whatnot is that someone should be yellin…

For those who wanna hear it https://youtu.be/-rqL035klC4?t=101

(Lightning at 1:03)

Re: Operational issue – Multiple services (UAE)

#56

Post by vntok »

jiggawatts wrote: post #48
bgentry wrote: post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc… Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances. A bit bored and annoyed, I pointed…

Should have pushed it.

Re: Operational issue – Multiple services (UAE)

#57

Post by stonogo »

ok_dad wrote: post #51
flymasterv wrote: post #50

Earlier quoted context omitted.

Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.

What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.

In that case, the hero was the person who avoided relying on a single AZ when they deployed to cloud.

Re: Operational issue – Multiple services (UAE)

#58

Post by pwarner »

Imustaskforhelp wrote: post #34
bgentry wrote: post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?

Re: Operational issue – Multiple services (UAE)

#59

Post by userbinator »

toast0 wrote: post #39
Imustaskforhelp wrote: post #23

Has this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was cause…

Historically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding. Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targette…

Cold War era definitely resulted in a lot of comms infrastructure being hardened against attack.

Re: Operational issue – Multiple services (UAE)

#60

Post by crote »

SoftTalker wrote: post #43
Imustaskforhelp wrote: post #23

Has this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was cause…

Data centers are usually built to withstand local natural risks e.g. weather. All bets, SLAs, and insurance are usually off when it comes to acts of war.

There's also just an upper limit to the kind of risk you can reasonably defend against.

An out-of-control wildfire levels the entire city? The Big One hits the Bay Area? The entire city is flooded for a few months because the levees break during a Cat5 hurricane? Yeah, your DC will be completely ruined. And even if it isn't, you're probably not getting any outside power, generator fuel, or repair technicians for a while.

No matter how much money you pump into hardening your own super-bunker DC, there will always be disasters you aren't prepared for. At a certain point it just makes more financial sense to abandon the idea of invulnerability and build a redundant site a few states over. Accept that you will occasionally lose one, and only protect against incidents where mitigation is cheaper than occasionally rebuilding.

Post reply on HN