Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

111–120 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#111
post #73

Earlier quoted context omitted.

A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that boug…

There was also a failure mode in Seagate drives where the bearing increased in stiction. As long as it was spinning, there was no problem. But if you spun it down, it might not spin up again. If you had a group of disks powered up for a long time, many could fail together at the next power cycle. A chaos monkey that randomly powers down disks one at a time can prevent this.

[deleted]

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#113
>The sub result for 4 simultaneous drive failures in 156 hours = 1.89187284e-13. That means the probability of it NOT happening in 156 hours is (1 – 1.89187284e-13) which equals 0.999999999999810812715 (12 nines).

Minor nitpick, this ignores the possibility of more than 4 failures, although this error only affects the fourth digit after the nines. Much more egregious is the following:

>there are 56 “156 hour intervals” in a given year

This is too simplistic, there are in fact infinitely many 156-hour intervals in a year, some of them just happen to overlap. This overlap can't simply be ignored because even if none of their 56 disjoint intervals contain 4 events this does not rule out the possibility of there being 4 events in some 156 hour interval they didn't take into account. In fact failing to take into account even one of the infinitely many intervals creates a blind spot (consider what happens if the drives happen to fail precisely at the start and end of a particular interval). You can still get a lower bound by e.g. ensuring none of the 56 intervals contain more than 1 failure, or by adding more intervals and ensuring none of them have more than 2 failures etc.

Their binomial calculation contains the same mistake.

A quick improved lower bound can be obtained by calculating the probability that any failure is followed by (at least) 3 other failures within 156 hours. For one failure this probability is given by the Poisson distribution and is

    Pc = 1 -\sum_{k
Now we get into some trouble because the failures and the probability of a 'catastrophic' failure are dependent, however the probability that any particular failure turns catastrophic is constant, so the expected number of catastrophic failures can't be greater than the expected number of failures times that constant, this gives a lower bound of

    Pc (365·24·λ) = 6.63154e-9
this is a lower bound, but that's still three fewer nines left than their claim.

Anyway let's just hope their data centres are more reliable than their statistics.

Edit: This last calculation can be justified by noting that the probability that 1 critical failure starts in a particular time interval is Pc times the probability of 1 failure in that interval plus some constant times the probability of more than one interval. Similarly the probability of more than one critical failure is at most the probability of more than one failure.

Now the probability of more than one failure in a time interval is dominated by the length of the interval, therefore if you calculate the density those parts fall away and you're left with a density of Pc λ critical failures per hour.

This seems to be an exact expression for the expected number of critical failures, and not just a lower bound. Although it is still a lower bound for the probability of a critical failure, albeit a fairly tight one.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#114
post #40
post #31

Earlier quoted context omitted.

Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.

It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.

The Crucial M4 SSD had a firmware bug [1] that caused it to fail repeatedly starting after 5,184 power on hours.

Disk failures could also be triggered by datacenter environmental factors shared among many drives like temperature or noise.

[1] https://www.storagereview.com/node/2676

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#115
post #40
post #31

Earlier quoted context omitted.

Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.

It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.

[deleted]

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#116
post #2

This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…

For a consumer the cheapest and easiest way to backup important documents or files is to encrypt it and store it across multiple storage providers, e.g. Dropbox and Google Drive. They usually give you a reasonable amount of free storage, and it's unlikely all accounts would be terminated or locked at the same time. And of course, you should always have your local backups as well.

Assume you want access to files over the next 20 years. What are the odds Google will have bought Dropbox in that time; and what are the odds an automated system monitor at Google ad words will have disabled your Google account in that time span?

Replace with Amazon and/or crashplan as appropriate..

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#117
post #56
post #35

Earlier quoted context omitted.

I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.

That's one of the reasons to encrypt locally and only store encrypted data on backup services.

If Google suspends your account for background music playing in a YouTube video, you might still lose access to your files in Google drive / cloud - even if the files are encrypted.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#118

Earlier quoted context omitted.

It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.

I tend to assume pessimistically that the durability design will itself cause a problem. Redundant switches to survive a hardware failure, e.g., strikes me as inviting trouble.

Indeed. I once had critical systems routed via a large redundant Cisco switch which claimed to be 1+1. Turns out there was a single "supervisor" component which failed (after just a year or two) and made the pair of switches useless. Apparently the designer worked in a team where nobody does anything when the boss is out.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#119
post #51
post #35

Earlier quoted context omitted.

I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.

Forget account ban, there is a nonzero risk a false positive of the automated kiddie porn search all the cloud storage providers do has your home searched and puts you in handcuffs.

While I'm sceptical of content filters, even with a home search, it seems unlikely you'd end up in cuffs unless a) the filter caught acymtual illegal content, or b) the search turned up something illegal.

You might get killed in the course of the initial police raid though..

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#120
post #98
post #95

Earlier quoted context omitted.

BB and S3 both have eleven 9 durability, how much does using both increase this?

Depends what you are modeling. The probabilities of random disk failures, are probably independent. However, there are risks that are not necessarily independent such as the US Government ordering these two services to delete your data, or, as the article mentions, an armed conflict destroying data centers.

Or your credit card being suspended and both services deny you access / delete your data.
Post reply on HN