Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

Post reply on HN
Print view View original

391–400 of 463 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#391

Post by eptcyka »

rob74 wrote: post #296
sandeepkd wrote: post #241

Earlier quoted context omitted.

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

Especially with Amazon, who are well known for squeezing every last bit of profit from their employees, contractors etc., it doesn't really sound surprising. "Offsite backups?! Sure, you could have had that if you had found the right page in the AWS console and if you would have paid 50% extra!"

It all depends on how *off* off-site really is. You know, *off* isn’t a binary, it is a spectrum.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#392

Post by steveBK123 »

cmiles8 wrote: post #387
steveBK123 wrote: post #235

Earlier quoted context omitted.

There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle…

AWS has a history of making hand wavy explanations on the robustness of its infrastructure that were misleading at best. See comments about how regions are truly independent only for everyone to find out if us-east-1 goes down you could still be down even if you built in other regions. Those dependencies were not well documented and previously hand-waved away by AWS when it boasted about how its regions were truly in…

I suspect at this point, given the nature of the AWS org.. they actually don't even know themselves either.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#393

Post by nacnud »

jackb4040 wrote: post #60
rishikeshs wrote: post #41

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.

Cool! I have a question about lottery numbers..

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#394

Post by cowsandmilk »

cmiles8 wrote: post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

1. This wasn’t a strike on a single data center, it was strikes on many data centers.

2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#395

Post by alexpotato »

When doing Disaster Recovery (DR) planning as a SRE in the New York City area, I sometimes use the phrase "Hurricane Sandy 2" to describe an event so big that it knocks out the power/compute/etc for an entire region.

I may just start using AWS Bahrain 2 as an additional example.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#396

Post by [deleted] »

jackb4040 wrote: post #60
rishikeshs wrote: post #41

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.

[deleted]

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#397

Post by varispeed »

cowsandmilk wrote: post #394
cmiles8 wrote: post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

1. This wasn’t a strike on a single data center, it was strikes on many data centers. 2. Since that interview, AWS has started selling versions of their storage that isn’t redundant. It is not surprising that when AWS sells non-redundant storage that some data is not recoverable.

> This wasn’t a strike on a single data center, it was strikes on many data centers.

He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#398

Post by austin-cheney »

sandeepkd wrote: post #241
krick wrote: post #109

Earlier quoted context omitted.

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

That is why contracts are more than one page in length. The details matter. I can remember receiving a contract class in Afghanistan about something as simple as moving gravel. Yeah, use your imagination with that and whatever absurd cartoon like fantasy you could dream up regarding "moving gravel" is still probably less strange than the real events that occurred.

I don't write contracts for a living, at least yet, but my learning so far is:

* clear goals: where is the end point and what does the product look like once it gets there in all required details

* defined test criteria: this is where you get to sue when they fuck shit up

* measures: there must be predefined measures. These can be wildly unrealistic at the start and require changes as the work occurs, which is ok, but there must be defined performance criteria that all parties are held to before work completion. In other worlds this is rewarded with bonus targets and penalties

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#399

Post by nullbio »

rishikeshs wrote: post #41

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

Azure has outages every 6 hours. It's really pleasant.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#400

Post by everfrustrated »

gregates wrote: post #355
krick wrote: post #109

Earlier quoted context omitted.

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life. For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reco…

As far as I'm aware Backblaze stores data only within a single datacenter (for a given region). This likely made sense in their original business model of being "offsite" copy of data.

But it very much breaks down for B2 where they're now storing original data. I hope they rethink this model. You do get what you pay for. There's a reason they're cheap.

Post reply on HN