Live data from Hacker News

Operational issue – Multiple services (UAE)

health.aws.amazon.com

61–70 of 125 posts

Re: Operational issue – Multiple services (UAE)

#61
post #31

Earlier quoted context omitted.

That's so interesting. Are any of the US military (or other satellite state of the US) systems running in "normal" datacenters or do they have a few protected DoD datacenters in the US?

Found this relevant article: https://serverlift.com/blog/military-modular-data-centers/ (AWS Military Modular Data centers) I do think that though, atleast from the Anthropic decision prior, we know that Anthropic which was used by DoD should be on normal AWS datacenters. I am saying this because, Dod Threatened to force take the source code of Anthropic if they don't agree to aggregious demands so that means that th…

they have a press release stating that Claude is available on bedrock in secret regions

https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-be...

https://aws.amazon.com/federal/secret-cloud/

Re: Operational issue – Multiple services (UAE)

#62
post #39

Has this ever happened ever in history of Cloud providers before this because of war? They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was cause…

Historically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding. Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targette…

not even a decade ago for an example (NSVLTNMT):

https://en.wikipedia.org/wiki/2020_Nashville_bombing

Re: Operational issue – Multiple services (UAE)

#63
post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services. The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few data…

Notably they did have backups. As you would expect for a $135 billion dollar undertaking. It's just that restoring from a calamity tends to be time consuming (a key difference between failover and a backup).

Re: Operational issue – Multiple services (UAE)

#64
post #56

Earlier quoted context omitted.

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc… Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances. A bit bored and annoyed, I pointed…

Should have pushed it.

at The Planet in Dallas c. 2002 the EPO button was exposed with no cover, and in very very close proximity to the "Exit" button for the doors...

one day, a colo customer hit the wrong button on the way out, and uhh, there was an outage

Re: Operational issue – Multiple services (UAE)

#65
post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

I wouldn't risk it.

Both Americans and Israelis are known for double taps. Surely Iran can adapt their tactics too.

Re: Operational issue – Multiple services (UAE)

#66
post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

> we will ensure we restore power and connectivity safely this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost. IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the…

> this would require human intervention

Amazon has self-propelled robots that handle their logistics and fulfillment, don't they? Send in the robots.

Re: Operational issue – Multiple services (UAE)

#67
post #19

Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…

Data centers in space no longer look so unreasonable when the requirement is “redundancy against multi site bomb strikes mid op”. A little depressing when some pieces start to fit together.

Re: Operational issue – Multiple services (UAE)

#68
post #25

The important quote from the timeline: Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the…

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc… Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances. A bit bored and annoyed, I pointed…

Equinix in Sydney plonked 2 datacenters right on top of each other, and still insists that they are useful as redundant sites.

There was a locally very funny situation for a while when a tech influencer was insisting both equinix sites could be shut down by a single building collapse. He was wrong, but he wasn't so wrong that people shouldn't make better infrastructure decisions.

Re: Operational issue – Multiple services (UAE)

#69

Earlier quoted context omitted.

The more interesting question, is how many datacenters are just plonked next to a high-value military target? For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

Is there acrually some meaningful physical separation between military and civilian server deployments? We seem to be really bad at separating those two. For example Starlink is basically military infrastructure now, used to guide bombs.

Arpa net
Post reply on HN