Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

21–30 of 92 posts

Re: How multi-disk failures happen

#21

Post by gav »

wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

To add some extra emphasis to wazoox's point, RAID-6 is always a better choice than RAID-5.

If you're willing take a capacity hit for improved write performance, RAID-1+0 is great. Though you can only survive two disks failing if they are in different pairs.

You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations.

Re: How multi-disk failures happen

#22

Post by gav »

Ologn wrote: post #20
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed. Also, only one hot spare is in the set. Another cost saving. Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity. What…

Personally I've not had much luck with hot-spares, I'd prefer to have the spare in the array (in the case of RAID-6) so I can find out if there's a problem with that drive before it's the only thing standing in the way of total failure.

I'm a fan of the temperature@lert USB sensors; $130 gets you peace of mind: http://www.temperaturealert.com/

Re: How multi-disk failures happen

#23

Post by andrewvc »

And this is why, everywhere I've ever worked, I've had to say RAID is NOT backup. There are varying degrees of receptiveness to this, because actual backups are a giant pain in the ass, and have a very annoying lifecycle.

Re: How multi-disk failures happen

#24

Post by mtts »

sschueller wrote: post #4

I have had a multi-disk failure occur with a RAID-1 setup. Server was pre-built from a large vendor and worked fine until both disks failed at the exact same time (within minutes). Took the disks out to find that they had sequential serial numbers. Called vendor for replacement only to have them tell me that they had issues with that batch, yet did not make any attempt to inform me. Spent the day restoring from tape…

This is a problem even if you don't buy pre-built. You're going to be buying similarly specced drives at similar times and you're probably buying from vendors from the same rough geographical area so chances are you're buying drives from the same batch anyway.

It used to be worse: all the drives in a RAID setup had to have the exact same specifications or the thing wouldn't work, which pretty much guaranteed near simultaneous failure of multiple drives, but even today, with somewhat more flexible software raid setups, it's still a problem.

At a place I used to work we used to joke that a drive failure warning from a RAID controller was nothing more than a signal to get out the backup tapes and start building a new server.

Re: How multi-disk failures happen

#25

Post by StavrosK »

martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure).

The article assumes no scrubbing, which is a stupid thing to run without, as detailed from the article. So it's basically "why pointing a gun at your foot and pulling the trigger is bad", "because you're going to shoot yourself in the foot".

Re: How multi-disk failures happen

#26

Post by DanBC »

andrewvc wrote: post #23

And this is why, everywhere I've ever worked, I've had to say RAID is NOT backup. There are varying degrees of receptiveness to this, because actual backups are a giant pain in the ass, and have a very annoying lifecycle.

There are better reasons for RAID not being back than this obscure rare fault condition.

Re: How multi-disk failures happen

#27

Post by StavrosK »

scarmig wrote: post #6
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?

I'm using zfsonlinux on Ubuntu, and so far (months) it has been solid. Granted, it's just for media storage, but it's fine.

Re: How multi-disk failures happen

#28

Post by StavrosK »

insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

How does ZFS discover bit rot if you don't read the bits? I have mine scrubbing once a week, is that unnecessary?

Re: How multi-disk failures happen

#29

Post by Dylan16807 »

ak217 wrote: post #14
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

The Linux answer to ZFS is btrfs, and it's almost ready to go.

Call me when btrfs has parity.

Re: How multi-disk failures happen

#30

Post by Dylan16807 »

lloeki wrote: post #11
scarmig wrote: post #6

Earlier quoted context omitted.

ZFS is beautiful and wonderful, but my perhaps outdated understanding is that the Linux port isn't as stable or mature as it is elsewhere. Is that not true? Or is it that the advantages of ZFS are so great that people should switch to illumos or some BSD?

> the Linux port isn't as stable or mature as it is elsewhere. Which one: the FUSE one, or the native ZFS on Linux under CDDL?

I haven't used the FUSE one extensively but the native one kept leaking memory until the system was exhausted. (Specifically it did this whenever I messed with a large number of files, I'm sure it works fine for media storage as StavrosK says.)
Post reply on HN