Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

Post reply on HN
Print view View original

141–150 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#141

Post by mrb »

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

«The chances that you only have 16 good drives for a file is about (0.97^16 ∗ 0.03^4)∗20∗19∗18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(»

Your math is completely wrong. In reality 92% of customers suffer no data loss.

The chance of a file having 4 failed drives (16 good drives) is: .03^4 = 0.00008100%

The chance of a file being irrecoverable is the chance of having 4 or more failed drives: .03^4 + .03^5 + .03^6 + ... = 0.00008351%

The chance of a file being recoverable is: 1 - .03^4 - .03^5 - .03^6 - ... = 99.99992151%

The chance of a customer's 100k files all being recoverable is: (1 - .03^4 - .03^5 - .03^6 - ...)^100000 = 92.0%

Therefore only 8% of customers encounter one or more 8MB file that is irrecoverable.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#142

Post by Johnny555 »

jsjohnst wrote: post #138
fs111 wrote: post #72

Earlier quoted context omitted.

That map is incorrect: us-west-2a, 2b and 2c are not static names for the AZs. Every user gets their own mapping of which physical location is a, which one is b and which one is c. My us-west-2a may be your us-west-2c. They are not the same.

Yes, I know they pseudo randomize the allocations. That is irrelevant to my core point, that each AZ is not just separate networks in the same building or even adjacent buildings, but rather they are truly isolated by a non-insignificant distance of somewhere around 50mi on average.

It's a cool map, but I can't find any reference for the source of the data center locations.

AWS purposely doesn't publish that information, and while I can believe it's possible to crowdsource the data by doing a little sleuthing (or working for certain vendors), it's hard to trust the map without knowing the sources.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#143

Post by arminiusreturns »

jrockway wrote: post #33
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center. Your power outage causing 3% of drives to fail is just a subset of that.

The first Datacenter I ever brought up to T2 was tornado proof (former military base).

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#144

Post by mmt »

foobarian wrote: post #140
mmt wrote: post #84

Earlier quoted context omitted.

> Correlated failures are common in drives. This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible. Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given dis…

I've been doing a poor man's version of this for home videos using 3 hard drives and rsync. It's easy to replace a drive and they are not likely to go out at the same time. But one thing that bugs me is that unless the drive fails hard (e.g. noticed by SMART or unable to read at all) how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas…

> how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas in the background?

I assume you're talking about already-written sectors becoming unreadable or a similar failure. Unfortunately, I don't think you can. This is what I believe the "patrol read" feature of RAID cards is meant to address.

Fortunately, however, I don't believe there's evidence that if the data is readable, it would ever be different from what had been written, so comparison isn't needed. The main exception to this is the case of firmware bugs that return sectors full of all-zeros.

> Does that impact durability of the drives?

I haven't read the studies (from Google, mostly, IIRC) in a while, and I'm not sure if they've released anything lately for more modern drives [1]. However, I believe you'll find an occasional "patrol read" won't noticeably reduce drive life/durability.

[1] Especially for something like SMR, whose tradeoffs would seem particularly attractive for something like this archival-like use case.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#145

Post by riku_iki »

Dylan16807 wrote: post #130
riku_iki wrote: post #10

> if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year. Also curious what are the stats from other providers.

The raw number would imply that, but I'm pretty sure the math breaks down when you're storing 10 byte records. Chunks of your data are going to be stored together, so it's a very small chance of losing a big block of 10 byte files. There's no failure mode that loses just one, and does so often.

> Chunks of your data are going to be stored together

It probably depends on their infrastructure, e.g. if storage is something like cassandra, records would be evenly distributed by key hash.

I agree that numbers likely are not like that though, just wanted to demonstrate that such calculation approach can bring unexpected conclusions.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#146

Post by kijin »

not_kurt_godel wrote: post #133
Dylan16807 wrote: post #131

Earlier quoted context omitted.

Not a tornado, though. > There are always unforeseen and unforeseeable risks associated with any location. You can mitigate them but you can't claim X number of 9s for a single physical datacenter. What is X, here? I'm pretty sure I can claim 99% for a single datacenter.

What I meant is you can't necessarily amortize loss in the event of a localized catastrophe. Failure modes in a single location are by definition not always statistically independent. You could have 99.99999% durability for 20 years, but if something happens to the datacenter that causes total loss, you're SOL. Geographical redundancy vastly reduces the risk of freak occurrences that you can't predict.

If a datacenter boasts flawless durability for 19 years and loses everything in the 20th year, then they have an infinite number of 9's for the first 19 years and zero for the 20th year. It's all about probability.

Nobody can promise 100%, but that doesn't mean that all those 9's are meaningless. They mean a lot for budgeting, and even more for insurance purposes -- which is exactly what we as a civilization have come up with as a way to amortize loss in the event of a local catastrophe. Your premiums are going to be much higher if you don't have enough 9's in a critical part of your money-making infrastructure.

No one here is saying that you don't need geographical redundancy. First we need to figure out how many 9's we can realistically expect in order to determine how much redundancy makes financial sense.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#147

Post by kalleboo »

the8472 wrote: post #122
e12e wrote: post #117

Earlier quoted context omitted.

If Google suspends your account for background music playing in a YouTube video, you might still lose access to your files in Google drive / cloud - even if the files are encrypted.

You should compartmentalize those.

That can be difficult to maintain though, as Google is pretty eager to link accounts.

You have horror stories like https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_gettin...

> Eventually someone realized that their non-work accounts were banned as well. It wasn't until yesterday that someone made the connection. Anyone who had their accounts as a recovery option were also caught in the ban wave.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#148

Post by kalleboo »

ujjain wrote: post #8

It still wouldn't upload my 1TB in back-ups in an entire month. Amazon Drive back-up completed in 3 days. Their pricing is amazing, but saving money on a back-up solution that doesn't seem as good as the other cloud storage providers is a dangerous game.

When I've gone a clean backup on Backblaze it's taken just over 24 hours to backup about 650 GB. And I'm not even in the US, so my data has to cross the pacific.

Backblaze is actually faster than Apple Time Machine is on my LAN which slightly bothers me. It also has lower CPU usage.

I originally chose Backblaze after benchmarking the other offers available at the time (Carbonite, Crashplan etc) and Backblaze was by far the fastest.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#149

Post by not_kurt_godel »

kijin wrote: post #146
not_kurt_godel wrote: post #133

Earlier quoted context omitted.

What I meant is you can't necessarily amortize loss in the event of a localized catastrophe. Failure modes in a single location are by definition not always statistically independent. You could have 99.99999% durability for 20 years, but if something happens to the datacenter that causes total loss, you're SOL. Geographical redundancy vastly reduces the risk of freak occurrences that you can't predict.

If a datacenter boasts flawless durability for 19 years and loses everything in the 20th year, then they have an infinite number of 9's for the first 19 years and zero for the 20th year. It's all about probability. Nobody can promise 100%, but that doesn't mean that all those 9's are meaningless. They mean a lot for budgeting, and even more for insurance purposes -- which is exactly what we as a civilization have com…

> No one here is saying that you don't need geographical redundancy

I mean, that's kind of what Backblaze is saying in the article, isn't it? They don't have geographical redundancy, yet there's not a single mention of that fact or the importance thereof in an entire article dedicated to teaching the unwashed masses about the limitations of mathematical theory in analyzing durability, even going so far as to say:

> somewhere around the 8th nine we start moving from practical to purely academic... it’s far more likely that...Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers [emphasis my own]

Seems like a pretty serious omission given their claimed authority as "the bottom line for data durability" and being "like all the other serious cloud providers" who do have geo redundancy, don't ya think?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#150

Post by [deleted] »

the8472 wrote: post #122
e12e wrote: post #117

Earlier quoted context omitted.

If Google suspends your account for background music playing in a YouTube video, you might still lose access to your files in Google drive / cloud - even if the files are encrypted.

You should compartmentalize those.

[deleted]
Post reply on HN