Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

11–20 of 92 posts

Re: How multi-disk failures happen

#11
post #6

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?

> the Linux port isn't as stable or mature as it is elsewhere.

Which one: the FUSE one, or the native ZFS on Linux under CDDL?

Re: How multi-disk failures happen

#12

isn't this what raid scrubbing is for? http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Data_Sc... for raid in /sys/block/md*/md/sync_action; do echo "check" >> ${raid} does that fix the issue? i run that once a week. i thought i was ok. am i not? if i am, isn't this old news?

yes, you should be fine. AFAIK most distros set up cronjobs for this automatically (for example, Debian/Ubuntu runs that once a month).

Re: How multi-disk failures happen

#13
post #6

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?

I haven't used the Linux version in a while, but as it's not a first class citizen of the kernel, I don't think it will ever be the same as a BSD implementation. In my opinion, if you're doing anything with storage, you'd be crazy not to use ZFS, even if that means learning a new OS/kernel.

Re: How multi-disk failures happen

#14

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

The Linux answer to ZFS is btrfs, and it's almost ready to go.

Re: How multi-disk failures happen

#15
Having worked at NetApp for a few years at the turn of the century this is "old" news :-) But it is always important to internalize this stuff. M3 (or 3x mirrors) is still computationally the most efficient (no compute just I/O to three drives), R6 is the most space efficient at the expense of some CPU. Erasure codes are great for read only data sets (they can be relatively cheap and achieve good bandwidth) but they suffer a fairly high I/O burden during write (n data + m code blocks for one read-modify-write).

Bottom line is that reliably storing data is more complicated than just writing it on to a disk.

Re: How multi-disk failures happen

#16

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I dealt with this exact problem for a number of years. Background scrubbing takes away I/O resources and can be a disaster on your workload if you rely on sequential reads/writes. For that reason, most controllers are configured by default to only scrub when the disk is totally idle which is never. Even if the controller had a better definition of idle, scrubbing an entire disk to find those rotten bits would take a long long time, a disk would almost certainly fail before that.

Re: How multi-disk failures happen

#17

If you follow the advice in this paper[1] you will be measuring media errors in your drives. That means re-reading all data every N days, even archived data. Without periodically re-reading and validating (checksumming) the data you can't tell if it has rotted in place. Since the distribution of errors over drives is very exponential you should then pro-actively remove the worst drives in your system. That will avoid…

Diamonds burn just as well as any other carbon does, once you get them hot enough.

De Beers probably like this usage of diamonds.

Re: How multi-disk failures happen

#18

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

On 3ware(LSI) controllers you can schedule a 'verify' task to run on a schedule. I do believe it is on by default, but I could be wrong. It is good to tell admins about this being that an uninformed one may turn the settings off without realizing what can occur if it is disabled.

Re: How multi-disk failures happen

#20
post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed.

Also, only one hot spare is in the set. Another cost saving.

Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity.

What if the disks are OK but the server host adapter card gets fried? Or the cable between the server and array? Some disk arrays allow for redundant access to the array, and some OS's can handle that failover.

Before I read the article, I thought it might discuss heat. Excessive heat is usually the cause when disk arrays start melting down one after another. Usually the meltdown happens in an on-site server closet/room which was never properly set up for having servers running 24/7. Usually the straw which breaks the back is a combination of more equipment added, and hot summer days. Then portable ACs are purchased to temporarily mitigate things, but if their condensation water deposits are not regularly dumped, they stop helping. This situation occurs more than you would imagine, luckily I have not been the one who had to deal with this for every time I have seen it (although sometimes I have). Usually the servers are non-production ones which don't make the cut to go into the data center.

The heat problem happens in data centers as well, believe it or not. A cheap thermometer is worth buying if you sense too much heat around your servers. Usually the heat problem is less bad, but the general data center temperature is a few degrees higher than what it should be, and this leads to more equipment failure.

Post reply on HN