Live data from Hacker News

AWS says it can't restore some data from mideast facilities struck by Iran

wsj.com

351–360 of 471 posts

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#351
post #110
post #85

Earlier quoted context omitted.

That's probably already true for 7 days or less

Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult. Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They di…

I need to bring this out:

https://craphound.com/overclocked/Cory_Doctorow_-_Overclocke...

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#352
post #109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

Well, listen, you do know that even when the largest most ambitious and most sophisticated things are shipped, each owner of each specific part (could be many many owners) basically , to the best of their ability, prayed that nothing particularly bad happens when it’s shipped off. Truly, that’s the best a mortal human can do, pray their part doesn’t break.

So then that big thing comes to you. It’s all kind of … held together by a prayer …

Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#353
post #330

11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquak…

> including acts of God Serious question: how do we know something is or isn't an act of god?

https://en.wikipedia.org/wiki/Act_of_God

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#355
post #109
post #73

This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…

That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.

It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life.

For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.

Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are sharded across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't shard across data centers at all, only across racks within a data center.)

Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.

Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40–50% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).

But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 has durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#356
post #296

Earlier quoted context omitted.

The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.

Especially with Amazon, who are well known for squeezing every last bit of profit from their employees, contractors etc., it doesn't really sound surprising. "Offsite backups?! Sure, you could have had that if you had found the right page in the AWS console and if you would have paid 50% extra!"

Perhaps, cheapness is always an factor but I'm potentially reading it as the customers perhaps not wanting data to move outside of the country and with AWS only have one datacenter in said country produced this result.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#357
post #139

Earlier quoted context omitted.

Why wouldn't they be able to collect? As long as Amazon does business in a country the legal system in that country can collect.

How would the Bahrain legal system force the USA to pay reparations for the war it started? Seize all US assets, the way we did with Russia?

I mean that is how it works in all legal systems. If you refuse to pay your assets will be seized. Which is why they would almost certainly not refuse.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#358

Earlier quoted context omitted.

It’s not really Amazon-caused

Yes it is, they didn't have adequate redundancy for this extremely foreseeable event (military site getting hit by missiles).

War?! In my Middle East? Preposterous.

Re: AWS says it can't restore some data from mideast facilities struck by Iran

#360

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop in…

> start talking about where administrative access and encryption keys lived.

As soon as you start specify technologies, rather than "sovereignty" you end up needing to created specific legal tests to stop people getting around it.

"Data must be stored domestically" is a short hand for being held in the same legal jurisdiction. This means for somewhere like the UK, you get all that battle tested data protections law for free. (new laws require case history to be reliable. Ie, prosecuting under a new law is hard, because if its on the edge of being legal, it can create a precedent that undermines the entire law)

In civil code places, its different, but I don't know enough to offer even a half arsed opinion.

The reason why jurisdiction is important is because if you are storing data outside of your legal protection, when something goes wrong there is little you can do to discourage fuckery.

This is the problem with blinkered engineering thinking. Yes geographically distributed data storage is good. But as you also know, storing it in place with lots of other data, means that its a target. The more places its stored, the more physical security you need. This means that there is higher chance of people being bribed.

Its not a binary, its a multi-dimension graph, with no one answer. Every dimension has a tradeoff.

UAE's tradeoff was: not even trump would ignore all the wargaming that clearly shows kicking iran in the nuts would have inflation rising consequences

Post reply on HN