AWS says it can't restore some data from mideast facilities struck by Iran • Hacker News Board

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#411

Post by TeMPOraL » Thu, Sep 17, 2026, 12:57 PM UTC

mitxela wrote: post #331

rcbdev wrote: post #305

Earlier quoted context omitted.

I just gave the crazy man a lot of money knowing he wants to start a holy war. I did not want him to fund this particular holy war that turned out to be disadvantageous to me!

Kind of like building your house on a floodplain and claiming insurance for a flood. They'd still say it was an act of god.

Well, flood is an act of God. "Floodplains" is another lie invented by those "scientists" to get you pissed, probably on a break from their usual attempts to convince you magnets are not a genuine miracle.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#412

Post by everfrustrated » Thu, Sep 17, 2026, 1:03 PM UTC

paulddraper wrote: post #377

ExoticPearTree wrote: post #282

Earlier quoted context omitted.

Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead. Now, there are a few things to consider: - AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data - You have to read the fine-print on what AWS offers…

All the AZs flooded simultaneously?! How big was this flood?

Azure and GCP definitions of AZ allow a single datacenter to have multiple AZs. AWS has by far the strictest definition.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#413

Post by AdamN » Thu, Sep 17, 2026, 1:04 PM UTC

jeffrallen wrote: post #383

purpleidea wrote: post #326

My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical…

I work on the same kind of thing, and while we think hard about bootstrap problems, we always find new surprising ones. The problem is you never know until you do it, and creating a faithful test of restarting giant systems is economically impossible. Because if you say to the boss, "look, I need 1 million now to test against a maybe 100 million loss, maybe in 10 years" they don't give you the money (and rightly so).

Even if you did the $1MM test there is very low likelihood that the $100MM event would be fully mitigated 10 years down the line (after who knows how many changes - physical, logical, and even in the org chart).

The only way to approach readiness here is repeated investment - like one team doing the deep dive and another pulling cables and then constantly doing pre- and post-mortems.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#414

Post by egorfine » Thu, Sep 17, 2026, 1:09 PM UTC

rishikeshs wrote: post #41

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

> data residency requirements in UAE

Is it a real law? I mean I would largely ignore those kinds of regulations on the basis of sheers stupidity. After all, lawmakers of the world tried to ban math for multiple times in the last few decades. Why would anyone consider conforming to laws like that? Especially since its not possible to enforce this law.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#415

Post by willturman » Thu, Sep 17, 2026, 1:20 PM UTC

jvm___ wrote: post #324

sandeepkd wrote: post #241

Earlier quoted context omitted.

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

It's funny because this applies to children cleaning home bathrooms as a Saturday chore as well.

Huh, I was thinking US Congresspeople.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#416

Post by torginus » Thu, Sep 17, 2026, 1:23 PM UTC

TeMPOraL wrote: post #410

paulddraper wrote: post #380

Earlier quoted context omitted.

Of course not. The surprising thing is that multiple availability zones were bombed simultaneously. And to my knowledge, they haven’t even yet said 2+ AZs were compromised.

Turns out that "availability zones" in cloud parlance seem to have nothing to do with geographical availability. Apparently they're logical splits and are more about billing than anything.

How often does an AZ go down while the other 2 still work?

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#417

Post by Betelbuddy » Thu, Sep 17, 2026, 1:24 PM UTC

mitxela wrote: post #362

Betelbuddy wrote: post #312

Earlier quoted context omitted.

>> AWS promises to keep data safe without requiring cross region redundancy. A perfect example of a claim they never made.

S3 is advertised as having 11 9's. That means in the entire history of S3 they've only lost a handful of objects, and will continue to lose objects at this same rate.

You are confusing the concepts, and applying the 11 nines to the wrong problem.

Durability is about: "If I successfully store an object in S3, how unlikely is S3 to permanently lose that object because of storage failures?"

It does not answer: "Will I be able to access that object, after a rain of Shahab-1 or Shahab-2 burn all data centers in the 3 availability regions across which my S3 bucket is spread out..."

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#418

Post by marcusverus » Thu, Sep 17, 2026, 1:24 PM UTC

krick wrote: post #109

cmiles8 wrote: post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The question is about one AWS facility. The headline refers to facilities. The guy's answer may well be an honest and truthful one.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#419

Post by ployable7 » Thu, Sep 17, 2026, 1:24 PM UTC

the8472 wrote: post #389

ployable7 wrote: post #203

Earlier quoted context omitted.

You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur. Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful t…

Seems like begging the question to me. You can't blend the numbers because amazon didn't blend the numbers. If they did and miraculously still arrived at 11 9s then that would also cover things such as wars and natural catastrophes, e.g. because they do offsite backups internally.

I’m not saying you can’t blend the numbers, I’m saying you shouldn’t blend the numbers. Because one number doesn’t communicate what you actually need to know to build.

You want to know how reliable the service is in steady state. For example it’s useful to know that S3 is effectively lossless in steady state whereas EBS volumes have an AFR of about 0.1%. You build your apps very differently between S3 and EBS knowing this. You can build highly resilient applications on each, but you code them differently, informed by these design goals.

You separately want to understand the failure modes that will require you to fully recover from backup. For example knowing that cloud storage is resilient to everything but region failure would inform you that your backups should be out of the region, not just a bucket in the same region. You don’t get that perspective from just a 9s number.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#420

Post by simoncion » Thu, Sep 17, 2026, 1:31 PM UTC

dannyobrien wrote: post #279

simoncion wrote: post #214

Earlier quoted context omitted.

> This is for data loss, of course, not access or surveillance or unlawful processing. This is why the minority of politicians who actually know about how this stuff works worry about where the data resides for jurisdictional purposes. If the government where the data resides can compel the folks who have physical and/or logical access to the physical machines that contain that data to give them access to that data,…

I mean, in those rooms I was arguing over the best policies to prevent access and surveillance and unlawful processing, and what the potential cost-benefit analysis was. And what I was arguing against was an assumption that physically compelling all companies -- or worse, all citizens -- to keep their data within the borders of the host country, would protect you from these problems. We'd have to explain that if the…

> But it's those politics and policies of the institutions with control over the data that are the most important part of this: not where the bits are kept. Especially when those bits are encrypted, and the keys and access controls are elsewhere.

Nah. Policies prohibit rule-followers from accessing data that you don't want accessed. Such policies are very important. But if you give your adversary effectively-unlimited physical access to the hardware where the bits are kept, that's game over. If you don't trust the governors of a region to honor the "don't tamper with this hardware" gentleman's agreement, and you very seriously care about preventing unauthorized access to the data that that hardware stores and processes, then you don't put that hardware in that region.

To point to a real-world example of this, there's not going to be an AWS Top Secret Cloud region in China, Russia, or -say- North Korea.