How multi-disk failures happen
sysadmin1138.net
How multi-disk failures happen
1–10 of 92 posts
Re: How multi-disk failures happen
#2If you follow the advice in this paper[1] you will be measuring media errors in your drives. That means re-reading all data every N days, even archived data. Without periodically re-reading and validating (checksumming) the data you can't tell if it has rotted in place. Since the distribution of errors over drives is very exponential you should then pro-actively remove the worst drives in your system. That will avoid an accumulation of errors and sudden multiple drive failure as described here.
Durability is like a diamond: it is forever.
Re: How multi-disk failures happen
#3I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get.
As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives.
The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about it. At the moment, the only real solution is ZFS.
Re: How multi-disk failures happen
#4I have had a multi-disk failure occur with a RAID-1 setup. Server was pre-built from a large vendor and worked fine until both disks failed at the exact same time (within minutes).
Took the disks out to find that they had sequential serial numbers.
Called vendor for replacement only to have them tell me that they had issues with that batch, yet did not make any attempt to inform me.
Spent the day restoring from tape backup.
TLDR: If you buy a pre-built server check that the disks aren't all from the same batch.
Re: How multi-disk failures happen
#5Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete, period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and all will be well.
I'd add that current "enhanced format" drives are tremendously better than most older drives. If your drive is a 1 or 2 TB old (512B sectors) drive, use it only for backup or whatever menial use for unimportant data.
Re: How multi-disk failures happen
#6I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…
ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?
Re: How multi-disk failures happen
#7I'm not a Unx sysadmin at all and don't know much about hard drive: I'm just a software dev.
But from the beginning of TFA, after reading this:
"Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected."
The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting such problem right away".
I've got a huge issue with the "largely undetected". I may, at one point, need storage for a gig I'm working on. And I certainly don't want stuff problems like that to go "largely undetected".
So quickly skipping most of the article and going to the comments:
"It's worth pointing out that many hardware RAID controller support a periodic "scrubbing" operation ("Patrol Read" on Dell PERC controllers, "background scrub" on HP Smart Array controllers), and some software RAID implementations can do something similar (the "check" functionality in Linux md-style software RAID, for example). Running these kinds of tasks periodically will help maintain the "health" of your RAID arrays by forcing disks to perform block-level relocations for media defects and to accurately report uncorrectable errors up to the RAID controller or software RAID in a timely fashion."
To which the author of TFA himself replies:
"Yes, that is something I should have made clearer. This is the very reason that RAID systems have background processes that scan all the blocks."
Which leaves me all a bit confused about TFA, despite all the shiny graphs.
Basically, I don't really understand the premises of "bad blocks going largely undetected" in 2013...
Re: How multi-disk failures happen
#8Not very efficient but I try to avoid placing all volumes on a single raid set.
Re: How multi-disk failures happen
#9isn't this what raid scrubbing is for? http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Data_Sc...
for raid in /sys/block/md*/md/sync_action; do
echo "check" >> ${raid}
Re: How multi-disk failures happen
#10If you follow the advice in this paper[1] you will be measuring media errors in your drives. That means re-reading all data every N days, even archived data. Without periodically re-reading and validating (checksumming) the data you can't tell if it has rotted in place. Since the distribution of errors over drives is very exponential you should then pro-actively remove the worst drives in your system. That will avoid…
Diamonds burn just as well as any other carbon does, once you get them hot enough.