Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

101–110 of 457 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#101

Earlier quoted context omitted.

turns out reading comprehension skills have still not gotten better in the future

In the future, some companies begin to store their data on-premises away from big centralized datacenters. But many companies do not, due to costs and the general friction of changing how things are done. If OP tells me the name of his company I can hop in my time machine and tell him how it plays out.

MENA is the on-prem capital of the world, don't worry.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#102

Earlier quoted context omitted.

As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?

That would be a legal nightmare. They don't necessarily know what customers' data residency requirements are.

This. We have (well, had) customers running in me-south-1 and once the first AZ went down we wanted to proactively move their data to other regions even just as cold backups. But our legal department slapped that down pretty quickly.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#103
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

He forgot to add "if it's Multi-AZ" ;)

multi-AZ doesn’t help against multi-AZ drones :)

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#104
post #56

Earlier quoted context omitted.

Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries

Typically 300 miles geographically but could be hard in some Gulf States

In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all.

Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#105

Earlier quoted context omitted.

[flagged]

Poor Russia, forced to invade Ukraine! My heart bleeds for the Russian army, so cruelly dragged across the borders into another country and forced to fight its way out.

[flagged]

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#106
Uh.

Uh-oh.

It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.

The wording certainly seems very careful:

"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#107
post #83
post #43

Earlier quoted context omitted.

The SLA excludes force majeure.

Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.

Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#110
post #85
post #74

Earlier quoted context omitted.

> to-do list becomes securing weapons, vehicles and fuel. I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.

That's probably already true for 7 days or less

Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult.

Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th.

I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough.

[1] https://web.archive.org/web/20061206181753/http://interdicto...

Post reply on HN