Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

111–120 of 463 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#111
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

Isn’t the problem that multiple datacenters in one zone were blown up?

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#112

Earlier quoted context omitted.

As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?

Most likely, their own data residency terms prohibit this. It would be interesting to know if, when 2 out of 3 AZs got destroyed, customers got a heads up to move their data to a different region?

We received repeated, constant heads up to move our data by the first AZ much less second. The problem is that nobody is storing data in Bahrain unless there are data residency requirements for it.

nobody wakes up one morning and chooses to launch instances, CDN or S3 and would choose Bahrain as that without a requirement to, we were contractually and legally forbidden (in the middle as a vendor) to copy even encrypted data where we don't have the key out for redundancy, so the best we could do was tell our subcustomers to download all of their buckets to their office or some employee laptops at their office

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#113
post #3

They guaranteed 11 9's durability, didn't they? e: Yep https://aws.amazon.com/s3/storage-classes/

Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.

But do they have Act of Special Operation exemptions?

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#115
post #68

Earlier quoted context omitted.

Works unless local laws specifically block you doing that which they do for some classes of data in some countries. Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe .

Do cloud providers even share data center locations so you can assess the "far enough" bit yourself?

You usually get city level location information. Depends on your definition for 'far enough' if that works for you.

me-south-1 is about 250 miles away from me-central-1, but that's not far enough in this instance. Given that, I think city level location information should be good enough.

250 miles is pretty good for weather or not specifically targeted destruction (wildfire / industrial explosions / arson), but it's clearly not enough if your data is in a building targeted in a regional war. Assuming datacenters remain targets in wartime, I think it's fair to assume if one datacenter in any particular country is attacked, all the rest of the datacenters in that country are likely to be attacked, too. In that case, offline storage (tapes and things) in inconspicuous locations might be the way.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#116
post #78
post #56

Earlier quoted context omitted.

Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries

A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.

Do you think they could ask for a backup

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#117
post #66

Earlier quoted context omitted.

Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.

> the government requirements requires me to store data only in UAE!

... but not only in an Azure datacenter!

(The obvious option here might be an encrypted backup on some hard drives in a safe in a local office.)

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#118
post #18

No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.

If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.

Isn't S3 claiming eleven nines of data durability?

https://aws.amazon.com/s3/storage-classes/

"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."

I wonder if "can't restore some data" includes any S3 data?

I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#119
post #109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#120
post #104

Earlier quoted context omitted.

Typically 300 miles geographically but could be hard in some Gulf States

In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all. Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.

They're likely going to target two datacenters, not the datacenters + your medium sized company's office NAS and the safe in the office manager's home.

(Encryption handles confidentiality concerns.)

Post reply on HN