Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

171–180 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#171
"Our nines go to eleven"

Well written, but there are other significant risks like losing access credentials (e.g. a password stored only on one device that is destroyed in the same accident in which its only user, who remembered the password by heart, dies) or being hacked by someone who gains access to cloud storage and intentionally erases or corrupts data.

Specialization is good, but if Backblaze is strictly in the business of storing data on hard disks, who's going to help with designing and maintaining the reliable complete system on top of their service that users actually need?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#172

Earlier quoted context omitted.

Yes, I know they pseudo randomize the allocations. That is irrelevant to my core point, that each AZ is not just separate networks in the same building or even adjacent buildings, but rather they are truly isolated by a non-insignificant distance of somewhere around 50mi on average.

It's a cool map, but I can't find any reference for the source of the data center locations. AWS purposely doesn't publish that information, and while I can believe it's possible to crowdsource the data by doing a little sleuthing (or working for certain vendors), it's hard to trust the map without knowing the sources.

There’s a The Atlantic article[0] that explains how to source the data pretty easily if you are willing to put in the effort.

https://www.theatlantic.com/technology/archive/2016/01/amazo...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#173
post #158

Earlier quoted context omitted.

Oh you are right. The OP still has a slight error though. C(20,4) = 4845 possible combinations of failing drives (4 out of 20.) Therefore the chance a file is irrecoverable is: .97^16 × .03^4 × 4845 = 0.241% (not 0.3%) But the OP's conclusion is largely correct: every customer will have some irrecoverable files.

I think you're not considering that failed drives will be replaced and the data on them reconstructed from the other shards. This failure mode requires 4 of 20 drives to fail in such a short amount of time that reconstruction cannot be completed.

Yes, but this was the OP's scenario: what happens when 3% of drives all fail at the same time when the DC is powered back on.

Edit: actually the math is still wrong. The chance any 4 out of 20 drive is failing is: .03^4 × C(20,4) = .03^4 × 4845 = 0.392% — There is no need to multiply by .97^16 as the status of the other 16 drives is irrelevant.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#174

Earlier quoted context omitted.

I thought AZs were in the same physical location, just separate networks, no?

They are separate datacenters, I don't think they make any promises about how far they are apart from each other, but at least in some regions they are 10+ miles apart. The AWS Cloud infrastructure is built around AWS Regions and Availability Zones. An AWS Region is a physical location in the world where we have multiple Availability Zones. Availability Zones consist of one or more discrete data centers, each with re…

So, a single AZ could span across multiple physical locations, as well?

Am I reading that correctly?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#175

Earlier quoted context omitted.

It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.

Although Rumsfeld often gets credit for this statement, it has been around for a long time before him https://en.wikipedia.org/wiki/There_are_known_knowns

I think Rumsfeld gets credit for it because he was using it in the most degenerate, disingeuous form possible. Rather than guarding against legitimate concerns and pursuing actual handling of potential issues, he was just trying to rationalize continuing policies that were demonstrably counterproductive. It's one thing to say there might be factors we don't know. It's another to day that simply because there might be such things, we should dedicate significant resources and lives to blindly flailing away under the assumption that it will help. A presumption of unknown unknowns puts you in a position of not acting, normally. There is no way to know that you aren't exacerbating and making a problem worse if you know that little.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#176
post #159
post #144

Earlier quoted context omitted.

> how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas in the background? I assume you're talking about already-written sectors becoming unreadable or a similar failure. Unfortunately, I don't think you can. This is what I believe the "patrol read" feature of RAID cards is meant to address. Fortunately, however, I don't believe there's…

"I don't believe there's evidence that if the data is readable, it would ever be different from what had been written, so comparison isn't needed. The main exception to this is the case of firmware bugs that return sectors full of all-zeros." Comparison is needed to address misdirected writes and bit rot in the very least, see "An Analysis of Data Corruption in the Storage Stack" [1]. You can't count on your drive fi…

Thanks! I had either missed that paper or had taken away more of the message that these errors are more likely to be from events like misdirected writes, cache flush problems (hence the high correlation with systems resets and not-ready-conditions), and firmware bugs (on-drive and further up the stack), rather than bit-rot.

Still:

> On average, each disk developed 0.26 checksum mismatches.

> The maximum number of mismatches observed for any single drive is 33,000.

Considering the latter can represent 132K on a modern, 4K-sectored drive, that's a remarkable amount of data loss, enough to warrant a checksumming higher up (such as in the filesystem).. in theory [1].

However, the fact that this was NetApp using their custom hardware as the testbed makes me wonder if the data are skewed, and if the numbers would be nearly this bad from a more "commodity" setup, such as at Google. The paper alludes to this when referring to the extra hardware for the "nearline" disks, and I'm always suspicious of huge discrepancies in statistics between "enterprise" and other disks, even more so when there's a drastic difference in comparison methodology.

It would be interesting to see if there are any numbers for more modern drives, especially as the distinction between "enterprise" and "consumer" drives is disappearing, if only because demand for the latter is disappearing.

[1] In practice, an individual isn't aware of the 16.5K/132K loss risk, which is vanishingly small compared to other risks, anyway, and businesses don't tend to care and have survived OK anyway.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#177

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I recently looked into the research on this topic, and it absolutely agrees with you--failures are not independent. Papers of note:

https://www.usenix.org/legacy/event/fast08/tech/full_papers/...

http://www.cs.toronto.edu/~bianca/papers/fast07.pdf

Take note of section 5 in both papers for their statistical models and how they match with real-world data.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#178

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

Year 2038 seems a bit simple to test. Can you just set the BIOS date to January 19 2038 03:00:00 UTC and wait 15 minutes to see what happens?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#179

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I recently looked into the research on this topic, and it absolutely agrees with you--failures are not independent. Papers of note: https://www.usenix.org/legacy/event/fast08/tech/full_papers/... http://www.cs.toronto.edu/~bianca/papers/fast07.pdf Take note of section 5 in both papers for their statistical models and how they match with real-world data.

There was a statistics class in college that kicked my ass. I never quite understood how to determine if two variables were independent or dependent. You get a vastly different answer if you get it wrong.

I run into people all the time that seem to have the same problem, to the point that it makes me wary of any software developer putting forth numbers that seem fantastical.

My gut reaction is that if Backblaze wants to keep reporting their disaster preparedness numbers that they need the assistance of an actuary to calculate them.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#180
post #119
post #51

Earlier quoted context omitted.

Forget account ban, there is a nonzero risk a false positive of the automated kiddie porn search all the cloud storage providers do has your home searched and puts you in handcuffs.

While I'm sceptical of content filters, even with a home search, it seems unlikely you'd end up in cuffs unless a) the filter caught acymtual illegal content, or b) the search turned up something illegal. You might get killed in the course of the initial police raid though..

Well you take a picture of your kid in the bathtub, now who can tell the difference?
Post reply on HN