Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

121–130 of 456 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#121
post #72

[flagged]

[flagged]

You mean the same people who were told to keep the hostages until after Reagan got elected?

The same people who bought weapons sold by the Reagan administration to fund the Sandistas?

That’s some revisionist history you’ve got there.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#122

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop insisting on where the data resided for jurisdictional purposes and start talking about where administrative access and encryption keys lived.

[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#123
post #99

They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.

I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#124
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

He forgot to add "if it's Multi-AZ" ;)

The me-south-1 (Bahrain) Region was three AZs: https://docs.aws.amazon.com/global-infrastructure/latest/reg...

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#125

This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.

If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.

Ah yeah, the EU, a historically stable area.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#126

What a nightmare scenario to tabletop. How do you even begin to recover from something like this?

For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably

a) letting customers in other areas know that their data is backed up to another continent

b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"

For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#127
post #107
post #83

Earlier quoted context omitted.

Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.

Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?

I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.

The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#128

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

You can always selfhost

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#129
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

Isn’t the problem that multiple datacenters in one zone were blown up?

They had nine years and more money than god to build redundancy in an unstable region.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#130
post #119
post #109

Earlier quoted context omitted.

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The caveat is always "if you're using the service correctly " which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data ce…

The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS loses their data. It's stupid, but at the same time these beliefs are surprisingly wide spread.
Post reply on HN