Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

Post reply on HN
Print view View original

71–80 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#71

Post by ChuckMcM »

Johnny555 wrote: post #17
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that bought a filer with several hundred drives started seeing large numbers fail suddenly. Up to the point they failed, not any indication. But it was run time hours related and thus correlated to all drives that started running at the same time. If you had put those drives in different data centers on different racks on different servers it wouldn't have mattered if you started 20 at the same time to be your 'tome', when they started failing it was possible to lose them all in a week.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#72

Post by fs111 »

jsjohnst wrote: post #24
gm-conspiracy wrote: post #21

Earlier quoted context omitted.

I thought AZs were in the same physical location, just separate networks, no?

Nope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles. This doesn’t list all the locations, but is a good map to get an idea: https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...

That map is incorrect: us-west-2a, 2b and 2c are not static names for the AZs. Every user gets their own mapping of which physical location is a, which one is b and which one is c. My us-west-2a may be your us-west-2c. They are not the same.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#73

Post by tlb »

ChuckMcM wrote: post #71
Johnny555 wrote: post #17

Earlier quoted context omitted.

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that boug…

There was also a failure mode in Seagate drives where the bearing increased in stiction. As long as it was spinning, there was no problem. But if you spun it down, it might not spin up again. If you had a group of disks powered up for a long time, many could fail together at the next power cycle.

A chaos monkey that randomly powers down disks one at a time can prevent this.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#74

Post by Pissompons »

It's a really nice blog post but coming from Backblaze, it would have been nice if they wrote it _after_ bringing the Phoenix DC fully online. When Amazon or Google say 11 9s, I can believe it but Backblaze still only has a single datacenter for most data. All it takes is an earthquake.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#75

Post by fencepost »

One question since I know some of the Backblaze folks respond to these threads:

In addition to calling (and possibly getting blocked/ignored), does your customer service staff send text messages? I suspect that a big percentage of the phone numbers you have are for cell phones these days, and I see a lot less SMS spam than I do telemarketing. SMS would also allow you to get a bit of info visible to recipients (e.g. "Backblaze CC Expired") with more detail once a message is opened.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#76

Post by klodolph »

GauntletWizard wrote: post #13
klodolph wrote: post #7

Earlier quoted context omitted.

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.

The event that wiped out the Dinosaurs was 65 million years ago, but the Mesozoic era lasted for ~200 million years. Your point stands, though.

The question of how many data points to use is a subtle one, though. I can say that I picked one data point because I was lazy and doing back of the envelope math, which is reasonable because we can be somewhat assured that I didn't choose a number of data points that was convenient for the hypothesis.

But if you're choosing two data points, my question is... why two? If you are choosing whether or not to reply based on whether or not the second data point fits with the first, then you're introducing selection bias. The chance that the second data point disagrees with the first by at least as much as the 200 My interval disagrees with the 1/65 My rate is equal to 1-(exp(-65/200)-exp(200/65)) = 0.32, which is not especially high.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#77

Post by ChuckMcM »

brianwski wrote: post #36
ChuckMcM wrote: post #15

Earlier quoted context omitted.

I was going to post essentially the same thing, so here is an upvote :-) While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now). Still it does help you to understand the they take data avai…

Disclaimer: I'm the author of the blog post. :-) > some of the assumptions that Brian was tossing out are not good ones to make. We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumpt…

Hey Brian! For the record I love that you are transparent about your assumptions, it is really helpful. I have been in very very similar debates, both at NetApp and at Google of all places. Peter Corbett, the guy who invented the dual parity scheme that NetApp uses, wrote a similar analysis as well for Fast '04 [1].

As someone who likes to geek out on failure proof systems and perfectly secure systems, neither of which are attainable but can be asymptotically approached, I think you are seeing the "there is always one level deeper" kinds of discussions. Personally I think of them as endorsements because if the exceptions get too extreme (say 'what if an asteroid hits?') then you know you've got all the bases covered.

Its only a problem if the person analyzing the analysis finds something that you really did not even consider. Then it opens up an opportunity to look at the problem a whole new way.

[1] https://www.usenix.org/legacy/publications/library/proceedin...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#78

Post by klodolph »

tjoff wrote: post #62
klodolph wrote: post #7

Earlier quoted context omitted.

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.

That's why no sane entity would use one earth. Use two and you can quickly recover.

Our goal is N+2, that way when one Earth is down for planned maintenance, you can endure unplanned loss of a second Earth.

N, of course, is always 1.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#79

Post by metalliqaz »

tjoff wrote: post #62
klodolph wrote: post #7

Earlier quoted context omitted.

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.

That's why no sane entity would use one earth. Use two and you can quickly recover.

Recovery isn't quick at all unless you have developed the Ansible tech tree.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#80

Post by thaumasiotes »

klodolph wrote: post #7
garettmd wrote: post #2

This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data…

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.

> if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines

I don't think this is a useful definition of your yearly durability. If your data center is down for maintenance during a period in which it is guaranteed that nobody wants to access it, that doesn't reduce your availability at all -- if your only failure is an asteroid that kills all of your customers, it would be more accurate to say you have 100.000000% availability than 99.999999%.

Post reply on HN