Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
1–10 of 190 posts
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#2This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line:
Because at these probability levels, it’s far more likely that:
- An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data is deleted.
The point that once you get to a certain point of durability (at least as far as hardware/software is concerned) you're chasing diminishing returns for any improvement. But the risks that are still there (and have been big issues for people lately) are billing issues. I think it's an important point that the operational procedures (even in non-technical areas like billing and support) are critical factors in data "durability"
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#3This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…
I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters. At that point it doesn't seem important or worthwhile to consider this risk, because it's dwarfed by the others.
This seems like a form of the same argument, and I wonder where else this arises and how people describe it.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#4This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…
I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters.…
I think in other contexts, you might say that these signals are "below the noise floor."
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#5This was an interesting read from a technical point of view, but also well written and refreshingly transparent.
I found the discussion about why it doesn't matter when you start talking about 11 nines of reliability to be hilariously true.
At the end of the day we're still flawed humans living in a hostile universe, and no matter how foolproof we make the technology, there are some weaknesses that just can't be eliminated.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#6Earlier quoted context omitted.
I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters.…
I think in other contexts, you might say that these signals are "below the noise floor."
As an audio engineer (as well as a dev), I definitely agree with this use of that phrase.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#7This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…
I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines.
The point about billing is better, though.
My other concern is that a software bug, operator error, or malicious operator deletes your data.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#8It still wouldn't upload my 1TB in back-ups in an entire month. Amazon Drive back-up completed in 3 days.
Their pricing is amazing, but saving money on a back-up solution that doesn't seem as good as the other cloud storage providers is a dangerous game.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#9This analysis is simplistic.
Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc.
When calculating your chance of failure, you have to include that, or your result is bogus.
Eg. Model A of drive has a failure rate of 1% per year, but when failed the symptom is failure of the drive to spin up from cold, however if already spinning it will keep working as normal.
3 years later, the datacenter goes down due to a grid power outage and a dispute with diesel suppliers so the generators go down. It's a controlled shutdown, so you believe no data is lost.
2 days later when grid power is back on, you boot everything back up, only to find out that 3% of drives have failed.
Not a problem. Our 17 out of 20 redundancy can recover up to 15% failure!
However, each customers data is split into files around 8MB, which are in turn split into the 20 redundancy chunks. Each customer stores say 1TB with you. That means each customer has ~100k files.
The chances that you only have 16 good drives for a file is about (0.97^16 * 0.03^4)2019*18 = 0.3%
Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#10> if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file.
Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year.
Also curious what are the stats from other providers.