Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

Post reply on HN
Print view View original

31–40 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#31

Post by httpz »

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#32

Post by londons_explore »

ChrisLomont wrote: post #26
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

>The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-( You just made the same mistake you're criticizing. You assumed the 100K files were uniformly and independently spread. They're also likely clustered, and perhaps not even at the same data center. Given the variety of drives Backblaze uses, the drives are also not likely to all be the same model, so…

>You just made the same mistake you're criticizing

Yes. Even if you take it into account, the vast majority of customers will see data loss, assuming random (but not even) shard distribution.

I rounded and approximated to avoid explaining too many probability rules... As you can see, it gives the same result at the end.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#33

Post by jrockway »

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center.

Your power outage causing 3% of drives to fail is just a subset of that.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#34

Post by kraftman »

eloff wrote: post #5

This was an interesting read from a technical point of view, but also well written and refreshingly transparent. I found the discussion about why it doesn't matter when you start talking about 11 nines of reliability to be hilariously true. At the end of the day we're still flawed humans living in a hostile universe, and no matter how foolproof we make the technology, there are some weaknesses that just can't be elim…

It's 'foolproof', like dog proof but for fools.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#35

Post by brador »

Animats wrote: post #18

Financial failure or service shutdown by the provider is the highest risk for long term storage. The backup services CrashPlan, Dell DataSafe, Symantec, Ubuntu One, and Nirvanix all shut down. Nirvanix only gave two weeks notice for users to save their data.[1] [1] https://www.computerweekly.com/opinion/Nirvanix-failure-a-bl...

I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#36

Post by brianwski »

ChuckMcM wrote: post #15
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I was going to post essentially the same thing, so here is an upvote :-) While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now). Still it does help you to understand the they take data avai…

Disclaimer: I'm the author of the blog post. :-)

> some of the assumptions that Brian was tossing out are not good ones to make.

We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumptions for S3 for comparison, we're all ears.

> that Drive Savers will exist as a company 10 years from now

Absolutely true, this calculation is only good RIGHT NOW. For example, one of the things that came up internally was "well, when drives get more dense the rebuild time rises, so this calculation will no longer be accurate in two years". But at the same time, we have some additional tricks and optimizations to make which we have not done yet to cut the 6 day average drive rebuild time down to 3 days. Also, drives last us about 5 years, so your data will be migrated to totally new drives 5 years from now. Those drives will absolutely have a different drive failure rate (maybe higher, maybe lower) so the calculation will no longer have the same result 5 years from now.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#37

Post by catwell »

Johnny555 wrote: post #17
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

Multi-region redundancy was on the roadmap for their B2 offering 10 months ago, not sure if it has shipped yet.

https://news.ycombinator.com/item?id=15125643

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#38

Post by matt4077 »

schoen wrote: post #3
garettmd wrote: post #2

This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…

I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters.…

It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind.

Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#40

Post by kolpa »

httpz wrote: post #31
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.

It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.
Post reply on HN