Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

61–70 of 92 posts

Re: How multi-disk failures happen

#61

Post by cperciva »

3amOpsGuy wrote: post #31
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

ZFS is great, possibly the closest to perfection available at any price today. But theres a word beginning with "O" and ending with "racle", they are so focused on the short term buck they are massacring their potential revenues with their short sighted approach of keeping Solaris out of everyone's hands.

... which is why you should be using FreeBSD.

Re: How multi-disk failures happen

#62

Post by baruch »

mbell wrote: post #58
wayne_h wrote: post #54

Earlier quoted context omitted.

You could be having the TLER problem. WD green drives can take a long time to do error correction - this causes the raid to drop the drive as failed. This solution is to respond with an error quickly and let the raid fix it with parity.... but at least it doesn't drop the drive. Or buy more expensive drives. Wait isnt that the idea behind RAID: Redundant Array of Inexpensive Disks.. You might want to look at this...…

TLER really isn't a problem with software raid which the drobo and all other home NAS's that I know of use.

It may very well be the issue, if a disk takes a long time to recover from errors even a software RAID will throw the disk out or risk being the problem by itself. TLER will limit the damage.

Ofcourse, if the software was smart enough to know how to handle this properly and recover automatically it need not drop the disk completely from the array.

Re: How multi-disk failures happen

#63

Post by ars »

andrewcooke wrote: post #47
rryan wrote: post #39

Earlier quoted context omitted.

ZFS auto-heals on read so the files you touch regularly are fine. You do need to scrub regularly to prevent bit-rot across all your files. I go with once-a-week as well.

so how is that better than raid? it sounds identical to what is described in the article - the problem was not scrubbing (afaict).

ZFS is at the file level, not the disk level. So recovering from errors (i.e. rebuilds) is MUCH faster.

Re: How multi-disk failures happen

#64

Post by ars »

papercruncher wrote: post #16
martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I dealt with this exact problem for a number of years. Background scrubbing takes away I/O resources and can be a disaster on your workload if you rely on sequential reads/writes. For that reason, most controllers are configured by default to only scrub when the disk is totally idle which is never. Even if the controller had a better definition of idle, scrubbing an entire disk to find those rotten bits would take a…

I use the built in SMART full disk check. It's quite good at only reading when the disk is idle, and it checks the entire disk.

A quick self test every day for all disks, and a long (i.e. full read) self test once a week.

The RAID is then checked on top of that one a month (although that slows things down a bit).

Re: How multi-disk failures happen

#65

Post by baruch »

martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

The disks themselves (new ones at least) have a background scan process (called BMS or BGMS) which helps considerably. The one thing the disk can't do by itself is correct unrecoverable errors since by the definition the disk can't recover from them :-)

The combination of BMS and disk scrubbing at the RAID level should handle almost all of the issues that are pointed by the original post.

Though RAID scrubs can and do take a long time to complete, depending on the performance impact that you are willing to suffer on a continuous basis it can take a week or two to perform proper scrubbing.

Proper scrubbing would include not just reading the RAID chunk on a disk but to also read the other associated chunks from the other disks and verify that the parity is still intact. In RAID5 you will not be able to recover if the parity is bad as you won't know what chunk has gone bad.

I've been coding such systems for a while now and as a shameless plug would point to http://disksurvey.com/blog/ if there are things of interest I'd be happy to take requests and write about them as well.

Re: How multi-disk failures happen

#66

Post by micro-ram »

wayne_h wrote: post #54
lobster45 wrote: post #49

Earlier quoted context omitted.

I have an 8 bay Drobo Pro with eight 2 TB drives, and I have a drive go out every few months. Of course this is because we went cheap with WD Green drives. However the Drobo Pro offers to use two drives "as protection" so you can have two drives go out and it keeps going.

You could be having the TLER problem. WD green drives can take a long time to do error correction - this causes the raid to drop the drive as failed. This solution is to respond with an error quickly and let the raid fix it with parity.... but at least it doesn't drop the drive. Or buy more expensive drives. Wait isnt that the idea behind RAID: Redundant Array of Inexpensive Disks.. You might want to look at this...…

The reboot loop may be a bad unit. I just had a first edition 8 bay replaced free even though it was way out of warranty. Call them and ask nicely.

Re: How multi-disk failures happen

#67

Post by akg_67 »

VLM wrote: post #41

My favorite raid 5 failure mode is when a old hardware card fails and you have no stock spare and the maint contract was not renewed years ago and the cost of a new card is too much to be expensed and buying as a capital replacement will take a week or two minimum to be approved. And/or the card has been discontinued so you have to buy a used one from a shady foreign surplus dealer. I've seen too much of this type of…

You don't need to replace old/discontinued RAID card with the used one (assuming you mean same make and model). The new card from same manufacturer will work. The RAID card writes DCBs on disks that are read by the new RAID card upon replacement.

Re: How multi-disk failures happen

#68

Post by akg_67 »

mrb wrote: post #53

The author is wrong that "enterprise quality disks just plain last longer". CMU did a study on this topic on a population of 100 thousand drives, and found that enterprise-grade drives do not seem to be more reliable than consumer-grade drives. See the conclusion in: http://www.cs.cmu.edu/~bianca/fast07.pdf This legend must die. The author is also wrong when saying "a non-recoverable read error [is] a function of dis…

The CMU study is most probably flawed as it looked at hardware replacement records and didn't take into consideration the different usage and threshold for replacement between enterprise and consumer drives. Most enterprise drives are used in enterprise servers and storage systems that monitor drive errors closely using SMART. The threshold for drive errors is much lower with such systems and drives are replaced quickly. My company (storage system vendor) replaces disk even when the SMART alerts impending failure and doesn't wait for actual failure. This will come across in hardware replacement log as more frequent replacement of enterprise disks. The consumer disks are used in consumer systems. The consumers don't proactively replace disks, they wait until disk actually fails. This will show up in hardware replacement log as less frequent replacement of consumer disk.

I have actually used 'defective' enterprise disks in consumer systems for years after they were labeled defective by storage system vendors. About a decade ago, I used to buy such defective enterprise disks in bulk at auction from server and storage manufacturers and sold them as refurbished disks to consumers after testing.

Re: How multi-disk failures happen

#69

Post by mrb »

akg_67 wrote: post #68
mrb wrote: post #53

The author is wrong that "enterprise quality disks just plain last longer". CMU did a study on this topic on a population of 100 thousand drives, and found that enterprise-grade drives do not seem to be more reliable than consumer-grade drives. See the conclusion in: http://www.cs.cmu.edu/~bianca/fast07.pdf This legend must die. The author is also wrong when saying "a non-recoverable read error [is] a function of dis…

The CMU study is most probably flawed as it looked at hardware replacement records and didn't take into consideration the different usage and threshold for replacement between enterprise and consumer drives. Most enterprise drives are used in enterprise servers and storage systems that monitor drive errors closely using SMART. The threshold for drive errors is much lower with such systems and drives are replaced quic…

I fail to see your point about the threshold of replacement. Assuming that enterprise-class drives get replaced sooner because sysadmins monitor SMART, it is still widely acknowledged that SMART errors are a strong indicated that the drive will fail soon. For example the Google study on drive reliability showed this correlation on consumer-class drives [1] There is no reason to believe this correlation doesn't exist with enterprise-class drives (or else, what would be the point of SMART?). Therefore the replacement threshold is mostly irrelevant as the enterprise drive replaced due to SMART would have failed soon anyway.

I really don't understand this skewed perception of consumer- vs enterprise-grade harddrives. Do you believe that enterprise CPUs are more reliable than consumer CPUs? How about enterprise NICs vs consumer NICs?

Consumer-grade drives are sold in volumes so much larger than enterprise-grade drives, that vendors have strong incentives to make them as reliable as possible. I would even say they have incentives to make them more reliable than enterprise-grade drives. Because a single percentage point improvement in their reliability will drastically reduce the costs associated to warranty claims and repairs.

My own experience confirms the CMU study. I have worked at 2 companies selling each about 2-5 thousand drives as part of appliances, to customers across the world. One company was using SCSI drives, the other IDE/SATA. And the replacement rates were similar.

I can see your point about the usage being different which could invalidate the CMU findings about consumer vs enterprise drive reliability. But I don't personally believe it explains it. The CMU study + my annecdotal evidence one 2-5 thousand drives + the fact that no study has ever showed data suggesting enterprise drives are more reliable, makes me think that they are not.

[1] http://static.googleusercontent.com/external_content/untrust...

Re: How multi-disk failures happen

#70

Post by jzwinck »

meaty wrote: post #37

Let us not forget the little considered other cause which usually takes out your tape infrastructure as well: fire. One outfit I worked decided to stick a brand new APC UPS in the bottom of the rack as it was in their office. It promptly caught fire and burned the entire rack out. The fire protection system did fuck all as well other than scare the shit out of the staff. Scraping molten cat5 cables off with a paint s…

Do you know what caused the fire? Improper installation? Overheating? Defective equipment?
Post reply on HN