Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

41–50 of 92 posts

Re: How multi-disk failures happen

#41

Post by VLM »

My favorite raid 5 failure mode is when a old hardware card fails and you have no stock spare and the maint contract was not renewed years ago and the cost of a new card is too much to be expensed and buying as a capital replacement will take a week or two minimum to be approved. And/or the card has been discontinued so you have to buy a used one from a shady foreign surplus dealer. I've seen too much of this type of thing... I can tolerate the (minimal) cost of software raid but I can't survive the possible downtime of hardware raid, so its been software raid for me for pretty much the last decade.

Another fun one was the quad redundant power supply with all four plugs going into the same power strip.

And there was the power supply that blew out every drive in the box simultaneously. I suppose its not any worse than a lightning strike, other than the onsite tech assumed it was a cooling failure, so he replaced the dusty fans and all the drives, thus destroying an entire set of drives upon powerup (and fans, I would guess).

The poison drive tray where every slot you jammed it into, it bent the backplane pins. That turned into a huge expensive disaster.

Re: How multi-disk failures happen

#42

Post by lysol »

gav wrote: post #21
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

To add some extra emphasis to wazoox's point, RAID-6 is always a better choice than RAID-5. If you're willing take a capacity hit for improved write performance, RAID-1+0 is great. Though you can only survive two disks failing if they are in different pairs. You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations.

And of course, depending on the application, you should be using 1+0 for performance reasons over 5 anyway, if it's a database server.

Re: How multi-disk failures happen

#43

Post by jrockway »

StavrosK wrote: post #25
martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure). The article assumes no scrubbing, which is a stupid thi…

The article describes why scrubs don't happen often enough: it's slow and disruptive. I have a 3-way RAID-1 /home partition (long story) and it's checked on the first Sunday of the month. I always remember this because I can tell from the performance of my workstation that something is up with the disk. This is with operations like a single thread running "ls". If you're running a production service, you're also going to notice, and you're also going to have more than 3TB of drive to scan. That makes running regular scans rather difficult.

Re: How multi-disk failures happen

#44

Post by caw »

wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

EMC uses RAID5 as the default for storage arrays, and then has some number of global hot spares. Netapp uses RAID6 by default, and then also has some number of hot spares. I've never had data loss from either system as a result of multi-drive failure. RAID5 is perfectly fine in most instances.

Desktop drives will drop out of RAID arrays frequently, so you have to use RAID6 if you choose to go that route. If a disk drops into deep checking mode for physical errors, then it won't respond to the RAID controller fast enough, and then be considered a dead drive. It will subsequently be re-detected, and then array has to be rebuilt.

Re: How multi-disk failures happen

#45

Post by mprovost »

Ologn wrote: post #20
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed. Also, only one hot spare is in the set. Another cost saving. Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity. What…

Hard drives are pretty resilient to high temperatures. Google did a reliability analysis of thousands of hard drives and found:

"Overall our experiments can confirm previously reported temperature effects only for the high end of our temperature range and especially for older drives. In the lower and middle temperature ranges, higher temperatures are not associated with higher failure rates. This is a fairly surprising result, which could indicate that datacenter or server designers have more freedom than previously thought when setting operating temperatures for equipment that contains disk drives. We can conclude that at moderate temperature ranges it is likely that there are other effects which affect failure rates much more strongly than temperatures do."

http://research.google.com/archive/disk_failures.pdf

Re: How multi-disk failures happen

#46

Post by mprovost »

insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

That's only true for open source implementations (if you can call ZFS that, the only really decent implementation is still locked up inside Oracle). Netapp's WAFL (which ZFS copied) is the original implementation of a RAID system that is resilient to bit rot and the other array vendors (EMC, HDS etc) all have similar systems.

Re: How multi-disk failures happen

#47

Post by andrewcooke »

rryan wrote: post #39
StavrosK wrote: post #28

Earlier quoted context omitted.

How does ZFS discover bit rot if you don't read the bits? I have mine scrubbing once a week, is that unnecessary?

ZFS auto-heals on read so the files you touch regularly are fine. You do need to scrub regularly to prevent bit-rot across all your files. I go with once-a-week as well.

so how is that better than raid? it sounds identical to what is described in the article - the problem was not scrubbing (afaict).

Re: How multi-disk failures happen

#48

Post by DanBC »

mprovost wrote: post #45
Ologn wrote: post #20

Earlier quoted context omitted.

Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed. Also, only one hot spare is in the set. Another cost saving. Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity. What…

Hard drives are pretty resilient to high temperatures. Google did a reliability analysis of thousands of hard drives and found: "Overall our experiments can confirm previously reported temperature effects only for the high end of our temperature range and especially for older drives. In the lower and middle temperature ranges, higher temperatures are not associated with higher failure rates. This is a fairly surprisin…

I'd love to see the same research for SSDs.

Also, parent post does talk about higher end temps, not middle range temps.

Re: How multi-disk failures happen

#49

Post by lobster45 »

wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

I have an 8 bay Drobo Pro with eight 2 TB drives, and I have a drive go out every few months. Of course this is because we went cheap with WD Green drives. However the Drobo Pro offers to use two drives "as protection" so you can have two drives go out and it keeps going.

Re: How multi-disk failures happen

#50

Post by wayne_h »

I have been doing raid data recovery for many years. A very common scenario is "Two Drives Failed at Once". This is usually not the case. What usually happens is 1 drive fails. The raid then goes into degraded mode and continues to function. Nobody notices the warnings. Some time later... months or years even, a second drive fails, the raid goes down - now they notice. They call in the techs who declare 'two drives failed'. This is when your data is most at risk. People start swapping boards and drives, repower the system, rebuild drives, rebuild parity, force stale drives online etc. I have seen alot of raids that would have been recoverable had they done the proper steps. Then they hand it over and say they found it this way... didn't touch a thing... www.alandata.com
Post reply on HN