Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

51–60 of 92 posts

Re: How multi-disk failures happen

#51

Post by SageRaven »

mprovost wrote: post #46
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

That's only true for open source implementations (if you can call ZFS that, the only really decent implementation is still locked up inside Oracle). Netapp's WAFL (which ZFS copied) is the original implementation of a RAID system that is resilient to bit rot and the other array vendors (EMC, HDS etc) all have similar systems.

What features are open source ZFS implementations (ZFS v28) lacking to that on Solaris aside from encryption? I run a 6-drive raidz2 array on my FreeBSD workstation, and I put it through its paces, and it's pretty awesome. Rock solid stable, in my experience.

I hear TRIM support for ZFS is now in the HEAD branch and in the works for FreeBSD 10.

Re: How multi-disk failures happen

#52

Post by archagon »

wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Maybe for the enterprise. What about home use?

About a month ago, I had 4 drives scattered around the house, each in its own enclosure, and I wanted to consolidate them into one unit. Money was an issue, so I wanted to recycle as many of them as possible instead of buying new ones. A Synology NAS along with a single extra drive allowed me near-optimal use of space with 1-drive redundancy. Of course, I have weekly backups to an external drive, so even if the array fails during a drive swap, I'll still have all my important files.

Any other solution would either require me to buy more drives (a significant expense at $100+ a pop), sacrifice redundancy, or build my own NAS with ZFS (which would have significant administration overhead, cost more, and be larger than my Synology unit).

Re: How multi-disk failures happen

#53

Post by mrb »

The author is wrong that "enterprise quality disks just plain last longer". CMU did a study on this topic on a population of 100 thousand drives, and found that enterprise-grade drives do not seem to be more reliable than consumer-grade drives. See the conclusion in: http://www.cs.cmu.edu/~bianca/fast07.pdf This legend must die.

The author is also wrong when saying "a non-recoverable read error [is] a function of disk electronics not a bad block". An NRE can happen for different reasons, one of them is when (data and error-correction) bits in the block get corrupted in a way that prevent the error-correction logic from detecting this error. So the block is technically bad, just not bad enough to cause the drive logic to declare it as a read failure.

Re: How multi-disk failures happen

#54

Post by wayne_h »

lobster45 wrote: post #49
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

I have an 8 bay Drobo Pro with eight 2 TB drives, and I have a drive go out every few months. Of course this is because we went cheap with WD Green drives. However the Drobo Pro offers to use two drives "as protection" so you can have two drives go out and it keeps going.

You could be having the TLER problem. WD green drives can take a long time to do error correction - this causes the raid to drop the drive as failed. This solution is to respond with an error quickly and let the raid fix it with parity.... but at least it doesn't drop the drive. Or buy more expensive drives. Wait isnt that the idea behind RAID: Redundant Array of Inexpensive Disks..

You might want to look at this... http://hardforum.com/showthread.php?t=1285254

btw, I am currently working on recovering an 8 bay drobopro thats in a reboot loop...

Re: How multi-disk failures happen

#55

Post by kristianp »

wayne_h wrote: post #50

I have been doing raid data recovery for many years. A very common scenario is "Two Drives Failed at Once". This is usually not the case. What usually happens is 1 drive fails. The raid then goes into degraded mode and continues to function. Nobody notices the warnings. Some time later... months or years even, a second drive fails, the raid goes down - now they notice. They call in the techs who declare 'two drives f…

Would have upvoted if not for the web link at the end of the comment.

Re: How multi-disk failures happen

#56

Post by dredmorbius »

ak217 wrote: post #14
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

The Linux answer to ZFS is btrfs, and it's almost ready to go.

What's it waiting on, and who other than Val is working on it?

Answering my own question: http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-progs....

... and Valerie Auror's less active on that than I'd thought. Hrm.

Re: How multi-disk failures happen

#57

Post by vacri »

thaumaturgy wrote: post #35
gav wrote: post #21

Earlier quoted context omitted.

To add some extra emphasis to wazoox's point, RAID-6 is always a better choice than RAID-5. If you're willing take a capacity hit for improved write performance, RAID-1+0 is great. Though you can only survive two disks failing if they are in different pairs. You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations.

> You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations. Right. Because it bears repeating still: RAID is not for backup, RAID is for high availability.

The simple way to support this is that RAID doesn't protect against a slip of the fingers on the commandline.

Re: How multi-disk failures happen

#58

Post by mbell »

wayne_h wrote: post #54
lobster45 wrote: post #49

Earlier quoted context omitted.

I have an 8 bay Drobo Pro with eight 2 TB drives, and I have a drive go out every few months. Of course this is because we went cheap with WD Green drives. However the Drobo Pro offers to use two drives "as protection" so you can have two drives go out and it keeps going.

You could be having the TLER problem. WD green drives can take a long time to do error correction - this causes the raid to drop the drive as failed. This solution is to respond with an error quickly and let the raid fix it with parity.... but at least it doesn't drop the drive. Or buy more expensive drives. Wait isnt that the idea behind RAID: Redundant Array of Inexpensive Disks.. You might want to look at this...…

TLER really isn't a problem with software raid which the drobo and all other home NAS's that I know of use.

Re: How multi-disk failures happen

#59

Post by mbell »

archagon wrote: post #52
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Maybe for the enterprise. What about home use? About a month ago, I had 4 drives scattered around the house, each in its own enclosure, and I wanted to consolidate them into one unit. Money was an issue, so I wanted to recycle as many of them as possible instead of buying new ones. A Synology NAS along with a single extra drive allowed me near-optimal use of space with 1-drive redundancy. Of course, I have weekly bac…

Backing up to an external drive isn't enough if your really worried about the data. If your house burns down, or the single backup drive fails, your out of luck.

Synology's devices support automatic backup to S3, use it.

Re: How multi-disk failures happen

#60

Post by Corrado »

Another way to think about this problem is to turn it on its head and assume that hard drives will fail and make it a strength and not a weakness. Something like OpenStack Storage[1] is built around the idea that hard drives are transient and replacing them should be painless. In fact, the more drives you have the less problems you have.

Basically, you keep multiple copies of the same data across different clusters of hardware. If a drive or two (or ten) go bad, just replace them, there is no rebuild time. Sure, it costs some disk space to keep n copies of data, but drives are just getting cheaper and there are de-duplication schemes being developed to help with this. Its not like RAID-6 is super efficient either.

Just my two cents...

[1] http://www.openstack.org/software/openstack-storage/

Post reply on HN