Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

81–90 of 92 posts

Re: How multi-disk failures happen

#81

Post by StavrosK »

olgeni wrote: post #79
StavrosK wrote: post #78

Earlier quoted context omitted.

Very nice, thank you, I will try that. I am rather dismayed, however, by learning in this thread that my disks have 4K sector sizes and ZFS autodetected 512 bytes, which means I'll have to destroy the pool and recreate it...

It happens all the time... If you run "camcontrol identify ada0" (or whatever your device is) you can find out before it is too late: sector size logical 512, physical 512, offset 0 This is from a lucky drive of course :)

Hmm, there's no such command in Ubuntu, maybe it's from BSD?

Re: How multi-disk failures happen

#82

Post by baruch »

CrLf wrote: post #76

One thing I've learnt early on my career as a sysadmin is that disk quality is very important, and so is the quality of the RAID controller or software RAID subsystem. After you have a multiple drive failure on a supposedly safe RAID-1, and get forced into stitching it back into operation with a combination of "badblocks" and "dd", you'll quickly understand why... A good RAID controller won't let a drive with bad sec…

Your advice is ok for someone who is willing to take no risks and to spend the money on that. It is not strictly correct for all situations. In fact storage arrays are not likely to drop a disk on the first medium error since medium errors are a fact of life and do not necessarily indicate a bad disk. Ofcourse, given that there is a medium error it warrants a long term inspection to make sure that the medium errors are not consistent and come too often on a specific drive, that is a cause of concern but a single medium error is of no real significance.

I also found that higher-end drives lie, I used SAS Nearline drives that failed easy and often and I used standard SATA drives that were more resilient. It depends on the vendor and make. May also depend on the batch but I never found a proof for that in my work.

Re: How multi-disk failures happen

#83

Post by CrLf »

baruch wrote: post #82
CrLf wrote: post #76

One thing I've learnt early on my career as a sysadmin is that disk quality is very important, and so is the quality of the RAID controller or software RAID subsystem. After you have a multiple drive failure on a supposedly safe RAID-1, and get forced into stitching it back into operation with a combination of "badblocks" and "dd", you'll quickly understand why... A good RAID controller won't let a drive with bad sec…

Your advice is ok for someone who is willing to take no risks and to spend the money on that. It is not strictly correct for all situations. In fact storage arrays are not likely to drop a disk on the first medium error since medium errors are a fact of life and do not necessarily indicate a bad disk. Ofcourse, given that there is a medium error it warrants a long term inspection to make sure that the medium errors a…

Maybe I was wrong in using the term "transient error"...

A bad block reallocation can be seen as a transient error from the controller's perspective, but it isn't silent provided the drive doesn't lie about it (and one would expect that a particular storage system vendor doesn't choose - and brand - drives that lie to their own controllers).

The storage system may ignore medium errors that force a repeated read (below a certain threshold), but they shouldn't ignore a medium error where the bad sector reallocation count increases afterwards (which is just another medium error threshold being hit, this time by the drive itself).

I'm not saying that higher-end drives are more reliable or not. Given that most standard SATA errors go undetected for longer, one could even argue that higher-end drives seem to fail much more frequently... I've had more FC drives replaced in a single EMC storage array than in the rest of the servers (which have a mix of internal 2.5in SAS and older 3.5in SCSI320 drives), and we certainly replace more drives in servers than desktops.

But that's another topic entirely.

Re: How multi-disk failures happen

#84

Post by andrewcooke »

ars wrote: post #63
andrewcooke wrote: post #47

Earlier quoted context omitted.

so how is that better than raid? it sounds identical to what is described in the article - the problem was not scrubbing (afaict).

ZFS is at the file level, not the disk level. So recovering from errors (i.e. rebuilds) is MUCH faster.

but that's not what we were talking about! no-one was saying "zfs sucks as much as raid but at least it rebuilds faster afterwards". the implication was that zfs avoided the problem in the article (when, it seems, both zfs and raid need to be scrubbed, and both avoid the problem when that is done).

Re: How multi-disk failures happen

#85

Post by olgeni »

StavrosK wrote: post #81
olgeni wrote: post #79

Earlier quoted context omitted.

It happens all the time... If you run "camcontrol identify ada0" (or whatever your device is) you can find out before it is too late: sector size logical 512, physical 512, offset 0 This is from a lucky drive of course :)

Hmm, there's no such command in Ubuntu, maybe it's from BSD?

camcontrol is from FreeBSD.

I don't have a Linux box available right now but maybe "hdparm -I" does something similar: "request identification info directly from the drive".

Re: How multi-disk failures happen

#86

Post by StavrosK »

olgeni wrote: post #85
StavrosK wrote: post #81

Earlier quoted context omitted.

Hmm, there's no such command in Ubuntu, maybe it's from BSD?

camcontrol is from FreeBSD. I don't have a Linux box available right now but maybe "hdparm -I" does something similar: "request identification info directly from the drive".

Yep, that works:

    Logical  Sector size:                   512 bytes
    Physical Sector size:                  4096 bytes
I'm guessing that's not very nice. My ZFS pool was created with ashift 9 (this is 2^9=512 bytes), when it should be 12 (2^12=4096). I will have to copy everything off and back on again.

For everyone who wants to check, and because I couldn't find info on it, run:

    zdb | grep ashift
And see if it's 9 or 12.

Re: How multi-disk failures happen

#87

Post by wazoox »

olgeni wrote: post #80
wazoox wrote: post #75

Earlier quoted context omitted.

Only when the array is mostly empty. For several years hardware RAID controllers have been rebuilding only used space, too. Really, ZFS isn't that much of the miracle some want it to be.

How do they know what space is used or unused?

Apparently they keep a block list somewhere.

Re: How multi-disk failures happen

#88

Post by ksec »

I am going to ask a honestly stupid question. What is going to happen to ZFS? Sorry if this is Slightly off topic. Although the comments has already started discussing on it.

The OpenSource version of it, or the BSD version of it is only up to v28. And it seems after that Oracle is no longer putting out update as open source and what will happen after that? Disparity between Oracle version and BSD version? And are features still being developed? Most of limitation listed in Wiki hasn't change at all for the past years and are still listed as under development.

Re: How multi-disk failures happen

#89

Post by corresation »

wazoox wrote: post #72
corresation wrote: post #36

Earlier quoted context omitted.

RAID-5 (or single parity RAID of any kind) is obsolete, period. RAID-6 offers different compromises relative to RAID-5 (for one, twice the parity space), so it isn't quite like one is the successor of the other. And once you're talking about multiple disk failures, you're at the existential point where you should probably be talking about whole array failures (e.g. your controller has quietly been writing junk for th…

> it isn't quite like one is the successor of the other. Given the current price of hard drives, I don't get how "twice the parity space" can even matter. Furthermore, modern RAID controllers perform almost exactly the same using RAID-5 or RAID-6 (verified on most 3Ware, LSI, Adaptec and Areca controllers). So yes, RAID-6 definitely is RAID-5 successor. > how to deal with that scenario. RAID is not an alternative to…

RAID-6 is in no universe a RAID-5 successor. Simplifications of enterprise needs and risk tolerances and compromise acceptance is sophistry. It is telling enough that despite the bluster of some on Hacker News, major storage vendors (ergo - people who know much more than you) still make RAID-5 the default. Maybe they just haven't read the news.

Regarding the backup -- yeah, no kidding. That was the point. If the argument is "this is better because it can accept one more of countless possible failure modes", then "better" can continue indefinitely (why not 10 parity copies?) In the real world of compromise considerations there is a benefit return assessment that draws a line at a probability point.

It also sounds like many on here think you buy a box of disks and then make one universal logical volume on it (e.g. "if you have a spare why not just make it RAID-6?"). Because the spare(s) are usually universal, and you have many logical volumes encompassing RAID-10, 0, 5, 6, whatever the situation calls for.

Re: How multi-disk failures happen

#90

Post by wglb »

cperciva wrote: post #61
3amOpsGuy wrote: post #31

Earlier quoted context omitted.

ZFS is great, possibly the closest to perfection available at any price today. But theres a word beginning with "O" and ending with "racle", they are so focused on the short term buck they are massacring their potential revenues with their short sighted approach of keeping Solaris out of everyone's hands.

... which is why you should be using FreeBSD.

For several reasons (now including coughunitycough), I wish I had built my in-basement cluster with freebsd rather than ubuntu. Likely I will move the boxes over one-by-one.
Post reply on HN