We have business in UAE. For whatever reason I defaulted to us-west-2 since these particular applications are not latency sensitive.
Operational issue – Multiple services (UAE)
21–30 of 125 posts
Re: Operational issue – Multiple services (UAE)
#22> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. God forbid we'd ever say that it was struck by a missile or a munition in an act of war.
Potential moon bear attack; we're waiting on satellite imagery to confirm it. https://youtu.be/pvjgIxuVdo4?t=96
Re: Operational issue – Multiple services (UAE)
#23Has this ever happened ever in history of Cloud providers before this because of war?
They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused.
Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows)
Had some ideas which show that this idea might be interesting, https://www.nature.com/articles/s44284-026-00406-2
I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
Re: Operational issue – Multiple services (UAE)
#24Re: Operational issue – Multiple services (UAE)
#25The important quote from the timeline:
Mar 01 9:41 AM PST
We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
Re: Operational issue – Multiple services (UAE)
#26Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…
> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.
The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers.
The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem.
Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund.
Relevant Kevin Faang video about it: https://www.youtube.com/watch?v=3GOAUyipnM4 [Google Accidentally Deletes $135 Billion Pension Fund, Chaos Ensues]
Re: Operational issue – Multiple services (UAE)
#27I wonder if this was bad targeting job or intentional. I appreciate the transparency and optimism in the status updates though!
Looking at Google Maps, there's Al Dhafra Air Base a couple of miles to the datacenter's south, an oil refinery a bit to the east, ports to the north, and a military academy to the west.
Re: Operational issue – Multiple services (UAE)
#28Re: Operational issue – Multiple services (UAE)
#29Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future. Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services. Again, this is just me thinking out loud on a…
> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.
Previous outage news makes it sound like the cloud providers still have quite a few logical single points of failure.
Re: Operational issue – Multiple services (UAE)
#30> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. God forbid we'd ever say that it was struck by a missile or a munition in an act of war.
Maybe a missile, maybe a drone, maybe debris
Doesn’t really matter, we know trumps latest war is the cause