Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter • Hacker News Board

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#11

Post by GuB-42 » Tue, Jul 17, 2018, 5:33 PM UTC

ujjain wrote: post #8

It still wouldn't upload my 1TB in back-ups in an entire month. Amazon Drive back-up completed in 3 days. Their pricing is amazing, but saving money on a back-up solution that doesn't seem as good as the other cloud storage providers is a dangerous game.

Not that dangerous if you do your own maths. I don't really trust cloud backup. As they said, the 11 9s doesn't matter. You are more likely to encounter a billing problem (as they said) but also get hacked, have a problem with your internet connection, many things can go wrong.

That's why you need another solution if you are serious about your data, maybe a set of external hard drives (local backup). This way, you have redundancy and little correlation in failure, which greatly improves your general durability. That local storage may be paid with the money you save by getting an "inferior" cloud backup provider.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#12

Post by londons_explore » Tue, Jul 17, 2018, 5:35 PM UTC

I've very disappointed their recovery time is 6 days!

Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines.

Why do you need it done in under a second? While the data is down one replica, it is at dramatically higher risk. Also, drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#13

Post by GauntletWizard » Tue, Jul 17, 2018, 5:36 PM UTC

klodolph wrote: post #7

garettmd wrote: post #2

This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.

The event that wiped out the Dinosaurs was 65 million years ago, but the Mesozoic era lasted for ~200 million years. Your point stands, though.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#14

Post by londons_explore » Tue, Jul 17, 2018, 5:38 PM UTC

I've very disappointed their recovery time is 6 days!

Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under 2 minutes. Maybe 10 minutes with some sluggish tail machines.

Why do you need it done so fast? Drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#15

Post by ChuckMcM » Tue, Jul 17, 2018, 5:43 PM UTC

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I was going to post essentially the same thing, so here is an upvote :-)

While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now).

Still it does help you to understand the they take data availability seriously which is the underlying message.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#16

Post by thefifthsetpin » Tue, Jul 17, 2018, 5:43 PM UTC

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I think that your point is what they were trying to address by saying that anything beyond eight nines is impractical. Their examples of correlated failures included earthquakes and floods which might have huge reach rather than just impacting a single rack, but I think it's the same general idea.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#17

Post by Johnny555 » Tue, Jul 17, 2018, 5:45 PM UTC

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

That could be a power surge taking out a whole rack

This failure mode, at least, is already accounted for by sharding data across cabinets:

Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet.

https://www.backblaze.com/blog/vault-cloud-storage-architect...

However, they don't seem to offer multi-datacenter (or multi-region) redundancy so are still susceptible to a datacenter fire/failure.

In comparison, AWS S3 distributes data across 3 AZ's (datacenters), and you can further replicate across regions if you choose. Though you pay for that added redundancy in 3 - 4X higher cost.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#18

Post by Animats » Tue, Jul 17, 2018, 5:48 PM UTC

Financial failure or service shutdown by the provider is the highest risk for long term storage. The backup services CrashPlan, Dell DataSafe, Symantec, Ubuntu One, and Nirvanix all shut down. Nirvanix only gave two weeks notice for users to save their data.[1]

[1] https://www.computerweekly.com/opinion/Nirvanix-failure-a-bl...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#19

Post by ChuckMcM » Tue, Jul 17, 2018, 5:48 PM UTC

londons_explore wrote: post #12

I've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines. Why do you need it done i…

Be careful here, it isn't 6 days until data is recovered it is 6 days until it is fully protected again, there is a big difference. During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly.

In most systems we assume that "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" which is recovering lost shards. So when you specify these things it is best to specify "how long to rebuild a shard while the array is providing storage services at its maximum specified rate." This assures the customer that if they have a 24/7/365 non-stop traffic pattern their data will still stay protected in the face of drive failures.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#20

Post by jsjohnst » Tue, Jul 17, 2018, 5:50 PM UTC

londons_explore wrote: post #12

I've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines. Why do you need it done i…

> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second.

I want to know where you can find a drive that can write 12TB/sec of data!

(In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code calculation is fast these days, but it ain’t that fast)