Earlier quoted context omitted.
A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that boug…
There was also a failure mode in Seagate drives where the bearing increased in stiction. As long as it was spinning, there was no problem. But if you spun it down, it might not spin up again. If you had a group of disks powered up for a long time, many could fail together at the next power cycle. A chaos monkey that randomly powers down disks one at a time can prevent this.
Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
111–120 of 190 posts
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#112Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#113Minor nitpick, this ignores the possibility of more than 4 failures, although this error only affects the fourth digit after the nines. Much more egregious is the following:
>there are 56 “156 hour intervals” in a given year
This is too simplistic, there are in fact infinitely many 156-hour intervals in a year, some of them just happen to overlap. This overlap can't simply be ignored because even if none of their 56 disjoint intervals contain 4 events this does not rule out the possibility of there being 4 events in some 156 hour interval they didn't take into account. In fact failing to take into account even one of the infinitely many intervals creates a blind spot (consider what happens if the drives happen to fail precisely at the start and end of a particular interval). You can still get a lower bound by e.g. ensuring none of the 56 intervals contain more than 1 failure, or by adding more intervals and ensuring none of them have more than 2 failures etc.
Their binomial calculation contains the same mistake.
A quick improved lower bound can be obtained by calculating the probability that any failure is followed by (at least) 3 other failures within 156 hours. For one failure this probability is given by the Poisson distribution and is
Pc = 1 -\sum_{k
Now we get into some trouble because the failures and the probability of a 'catastrophic' failure are dependent, however the probability that any particular failure turns catastrophic is constant, so the expected number of catastrophic failures can't be greater than the expected number of failures times that constant, this gives a lower bound of Pc (365·24·λ) = 6.63154e-9
this is a lower bound, but that's still three fewer nines left than their claim.Anyway let's just hope their data centres are more reliable than their statistics.
Edit: This last calculation can be justified by noting that the probability that 1 critical failure starts in a particular time interval is Pc times the probability of 1 failure in that interval plus some constant times the probability of more than one interval. Similarly the probability of more than one critical failure is at most the probability of more than one failure.
Now the probability of more than one failure in a time interval is dominated by the length of the interval, therefore if you calculate the density those parts fall away and you're left with a density of Pc λ critical failures per hour.
This seems to be an exact expression for the expected number of critical failures, and not just a lower bound. Although it is still a lower bound for the probability of a critical failure, albeit a fairly tight one.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#114Earlier quoted context omitted.
Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.
It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.
Disk failures could also be triggered by datacenter environmental factors shared among many drives like temperature or noise.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#115Earlier quoted context omitted.
Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.
It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#116This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…
For a consumer the cheapest and easiest way to backup important documents or files is to encrypt it and store it across multiple storage providers, e.g. Dropbox and Google Drive. They usually give you a reasonable amount of free storage, and it's unlikely all accounts would be terminated or locked at the same time. And of course, you should always have your local backups as well.
Replace with Amazon and/or crashplan as appropriate..
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#117Earlier quoted context omitted.
I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.
That's one of the reasons to encrypt locally and only store encrypted data on backup services.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#118Earlier quoted context omitted.
It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.
I tend to assume pessimistically that the durability design will itself cause a problem. Redundant switches to survive a hardware failure, e.g., strikes me as inviting trouble.
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#119Earlier quoted context omitted.
I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.
Forget account ban, there is a nonzero risk a false positive of the automated kiddie porn search all the cloud storage providers do has your home searched and puts you in handcuffs.
You might get killed in the course of the initial police raid though..
Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
#120Earlier quoted context omitted.
BB and S3 both have eleven 9 durability, how much does using both increase this?
Depends what you are modeling. The probabilities of random disk failures, are probably independent. However, there are risks that are not necessarily independent such as the US Government ordering these two services to delete your data, or, as the article mentions, an armed conflict destroying data centers.