Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

241–250 of 457 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#241
post #109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#242

Earlier quoted context omitted.

>Quite often a senior leadership issue because backing stuff up results in $$$ spent, which makes shareholders unhappy. This is true. The latter half of your comment is not. At best they have a duty to shareholders. But your assertion would mean every time a company posted a loss and the price went down the execs would be in legal trouble, which is nonsense.

Shareholder nuisance lawsuits can happen basically any time the stock goes down... or doesn't go up enough, and often companies will settle instead of the expensive fighting...

There are grounds legally if shareholders can prove that the "long-term risk reduction" argument is a lie, they can sue under the Duty of Loyalty. In reality, yes, this amounts nuisance lawsuits, large settlements and disruption at the board level. In practice, this and the executive pay structures almost always incentivizes short-term decisions and these risks become problems “for the next person.”

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#243
post #205

Earlier quoted context omitted.

Yep, exactly. There's a peculiarly unsatisfying kind of vindication that comes from making "slippery slope" arguments, and then watch them play, and now you are now both a) technically correct, and b) fucked. You'll excuse me if I have a brief "I told you so" moment about a scenario about Amazon's UAE datacenters being bombed without bakcups because of a US-instigated Iranian conflict, where -- if I'd ever dared to d…

I guess we aren't really close. You're saying (correct me if I'm wrong) that data sovereignty laws are unconditionally bad because they inevitably lead to totalitarian followon laws and there's nothing to stop them. I'm saying if we didn't have enforceable data sovereignty laws we would be in worse shape for data privacy and we should have prevented the followon laws from coming to be (and, true enough, we didn't). F…

[flagged]

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#244
post #109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

[deleted]

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#245
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle…

> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.

Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.

If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#247

Earlier quoted context omitted.

There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle…

> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc. Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering. If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it…

aws egress fees: the real hero we didn't know we needed

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#248
post #3

They guaranteed 11 9's durability, didn't they? e: Yep https://aws.amazon.com/s3/storage-classes/

Hot take: if you have a service that is 11 nines reliable, but there is an underlying component whose reliability is lower, cap the nines to that component.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#249
post #230

Earlier quoted context omitted.

The problem is that lots of people seem to be under the impression that they are doing it right because they are using AWS. They don't realize that AWS is a toolbox, not a 'ready made solution for redundancy against all catastrophes you are possibly exposed to'. They use that to their advantage by pricing such solutions at a level that people will either pay through the nose or will be left without recourse when AWS…

Yep, a lot of non-technical leaders believe that 'cloud' is synonymous with 'DR strategy' or even 'backup'. "We won't have to worry about being offline if our server goes down if we move to the cloud!" Some of these people fundamentally don't understand what the cloud is, their assumption is cloud means easy button that solves all your infrastructure and uptime problems.

I mean, can't cloud kinda solve all of those if you just use the right things?

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#250
post #169

Earlier quoted context omitted.

multi-AZ doesn’t help against multi-AZ drones :)

black swan events. this is one of one of those things - were in the current era either a cloud provider should provide automatic backups in another geographic zone. if you're in us-east, then your back-ups should ideally be in eu-west + africa for redundancy.

"Black Swan" would be a good name for an attack drone
Post reply on HN