Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter • Hacker News Board

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#161

Post by _urga » Wed, Jul 18, 2018, 8:24 AM UTC

brianwski wrote: post #82

ChuckMcM wrote: post #77

Earlier quoted context omitted.

Hey Brian! For the record I love that you are transparent about your assumptions, it is really helpful. I have been in very very similar debates, both at NetApp and at Google of all places. Peter Corbett, the guy who invented the dual parity scheme that NetApp uses, wrote a similar analysis as well for Fast '04 [1]. As someone who likes to geek out on failure proof systems and perfectly secure systems, neither of whi…

Good link! I've both forwarded it on, and will study it when I have some time later.

Hi Brian, thanks for open-sourcing Backblaze's JavaReedSolomon, which is really well-written. A few months ago I ran into an issue with Reed Solomon coding throughput not saturating the write throughput of 16 drives, and wrote a new Reed Solomon module based on Cauchy matrices: https://github.com/ronomon/reed-solomon

The Cauchy matrices remove the need for a table lookup to do the Galois multiply, replacing it with pure XOR. Together with other optimizations, this gives nearly 3x-5x more coding throughput for the same (17,3) parameters, assuming you're still using your open-sourced JavaReedSolomon in production. I don't know if Reed Solomon coding throughput is a factor in your rebuild times?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#162

Post by u02sgb » Wed, Jul 18, 2018, 8:33 AM UTC

atYevP wrote: post #105

fencepost wrote: post #75

One question since I know some of the Backblaze folks respond to these threads: In addition to calling (and possibly getting blocked/ignored), does your customer service staff send text messages? I suspect that a big percentage of the phone numbers you have are for cell phones these days, and I see a lot less SMS spam than I do telemarketing. SMS would also allow you to get a bit of info visible to recipients (e.g. "…

Yev from Backblaze here -> I believe we do send SMSs in the case of a Cap or Alert getting reached, so yes that could be possible - though I'm not sure if an SMS is part of our billing failure process - that's an interesting question!

Backblaze B2 customer here. My credit card stopped accepting your billing and no SMS for me. Took me a month or so to notice the emails and update my details. I've got SMS alerts active for Caps. Would be worth adding that as was a bit scary when I noticed the mail (think it was the third one you'd sent!).

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#163

Post by _urga » Wed, Jul 18, 2018, 8:40 AM UTC

pixl97 wrote: post #55

schoen wrote: post #3

Earlier quoted context omitted.

I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters.…

https://en.wikipedia.org/wiki/Black_swan_theory

https://en.wikipedia.org/wiki/Information_theory

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#164

Post by mijamo » Wed, Jul 18, 2018, 8:44 AM UTC

not_kurt_godel wrote: post #149

kijin wrote: post #146

Earlier quoted context omitted.

If a datacenter boasts flawless durability for 19 years and loses everything in the 20th year, then they have an infinite number of 9's for the first 19 years and zero for the 20th year. It's all about probability. Nobody can promise 100%, but that doesn't mean that all those 9's are meaningless. They mean a lot for budgeting, and even more for insurance purposes -- which is exactly what we as a civilization have com…

> No one here is saying that you don't need geographical redundancy I mean, that's kind of what Backblaze is saying in the article, isn't it? They don't have geographical redundancy, yet there's not a single mention of that fact or the importance thereof in an entire article dedicated to teaching the unwashed masses about the limitations of mathematical theory in analyzing durability, even going so far as to say: > s…

I don't think geo redundancy helps much. Your data is more likely to be corrupted by some software of the provider than some random storm, or by some common hardware used by the same provider,

If you need to be safe about your data, you NEED several cloud providers in different places, with different softwares and different countries.

Especially for data it is pretty easy to just back it in 2 really different places at different providers. Relying on the geo redundancy of ONE provider and having to pay for it seems a bit useless for me.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#165

Post by kijin » Wed, Jul 18, 2018, 8:45 AM UTC

not_kurt_godel wrote: post #149

kijin wrote: post #146

Earlier quoted context omitted.

If a datacenter boasts flawless durability for 19 years and loses everything in the 20th year, then they have an infinite number of 9's for the first 19 years and zero for the 20th year. It's all about probability. Nobody can promise 100%, but that doesn't mean that all those 9's are meaningless. They mean a lot for budgeting, and even more for insurance purposes -- which is exactly what we as a civilization have com…

> No one here is saying that you don't need geographical redundancy I mean, that's kind of what Backblaze is saying in the article, isn't it? They don't have geographical redundancy, yet there's not a single mention of that fact or the importance thereof in an entire article dedicated to teaching the unwashed masses about the limitations of mathematical theory in analyzing durability, even going so far as to say: > s…

As mentioned elsewhere in this thread, Backblaze is working on adding another datacenter.

Personally, I don't care whether a single provider has multiple datacenters or not, because I prefer to have redundancy across providers. But that's not the kind of recommendation that we're likely to see on the blog of one of those providers.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#166

Post by kdkeyser » Wed, Jul 18, 2018, 8:59 AM UTC

_urga wrote: post #161

brianwski wrote: post #82

Earlier quoted context omitted.

Good link! I've both forwarded it on, and will study it when I have some time later.

Hi Brian, thanks for open-sourcing Backblaze's JavaReedSolomon, which is really well-written. A few months ago I ran into an issue with Reed Solomon coding throughput not saturating the write throughput of 16 drives, and wrote a new Reed Solomon module based on Cauchy matrices: https://github.com/ronomon/reed-solomon The Cauchy matrices remove the need for a table lookup to do the Galois multiply, replacing it with p…

There is also Intel's ISA-L: https://github.com/01org/isa-l

It contains optimized Galois Field multiplication, resulting in Reed Solomon (both Vandermonde as well as Cauchy) at multiple GB/s on a modern x86 CPU.

Still, I doubt that the Reed Solomon coding speed is the limiting factor in their rebuild time. There is a mention of a 6-day duration, so even with a very slow Reed Solomon implementation ( ~ 100 MB/s) that should not be a bottleneck for a 10 TB drive rebuild (assuming a distributed rebuild approach, not a traditional RAID style rebuild).

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#167

Post by rendaw » Wed, Jul 18, 2018, 9:01 AM UTC

aikinai wrote: post #108

I really want to like Backblaze and they seem to do a lot of good work, but whenever this comes up, I also feel responsible to let people know the dark side so they're informed at least. I've written in more detail before[0], but just to share the gotchas in case anyone here is thinking of switching to Backblaze: 1. They backup almost no file metadata. 2. The client is very slow (days or more) to add new files and th…

As much as this is off topic, I'd like to continue this conversation. Do you have any details/reference for point 1?

I've been using rclone (got the recommendation here) which has been reliable.

Also, does anyone know if Backblaze has any plans to offer u2f? I've switched dns and email providers to get u2f.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#168

Post by [deleted] » Wed, Jul 18, 2018, 9:02 AM UTC

jsjohnst wrote: post #20

londons_explore wrote: post #12

I've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines. Why do you need it done i…

> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. I want to know where you can find a drive that can write 12TB/sec of data! (In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code…

[deleted]

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#169

Post by aikinai » Wed, Jul 18, 2018, 9:12 AM UTC

rendaw wrote: post #167

aikinai wrote: post #108

I really want to like Backblaze and they seem to do a lot of good work, but whenever this comes up, I also feel responsible to let people know the dark side so they're informed at least. I've written in more detail before[0], but just to share the gotchas in case anyone here is thinking of switching to Backblaze: 1. They backup almost no file metadata. 2. The client is very slow (days or more) to add new files and th…

As much as this is off topic, I'd like to continue this conversation. Do you have any details/reference for point 1? I've been using rclone (got the recommendation here) which has been reliable. Also, does anyone know if Backblaze has any plans to offer u2f? I've switched dns and email providers to get u2f.

Yeah, it's a bit old, but here's an article about Backblaze not supporting metadata. [0] "It fails all but one of the Backup Bouncer tests, discarding file permissions, symlinks, Finder flags and locks, creation dates (despite claims), modification date (timezone-shifted), extended attributes (which include Finder tags and the “where from” URL), and Finder comments."

And I don't know if it supports U2F, but it does support TOTP.

[0] https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-bac...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#170

Post by majewsky » Wed, Jul 18, 2018, 9:21 AM UTC

mrb wrote: post #158

barbegal wrote: post #154

Earlier quoted context omitted.

The original maths is correct. You have neglected the fact that the 4 failed drives can be any of the 20. Your calculations are for 4 specific drives failing.

Oh you are right. The OP still has a slight error though. C(20,4) = 4845 possible combinations of failing drives (4 out of 20.) Therefore the chance a file is irrecoverable is: .97^16 × .03^4 × 4845 = 0.241% (not 0.3%) But the OP's conclusion is largely correct: every customer will have some irrecoverable files.

I think you're not considering that failed drives will be replaced and the data on them reconstructed from the other shards. This failure mode requires 4 of 20 drives to fail in such a short amount of time that reconstruction cannot be completed.