Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

31–40 of 92 posts

Re: How multi-disk failures happen

#31

Post by 3amOpsGuy »

insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

ZFS is great, possibly the closest to perfection available at any price today.

But theres a word beginning with "O" and ending with "racle", they are so focused on the short term buck they are massacring their potential revenues with their short sighted approach of keeping Solaris out of everyone's hands.

Re: How multi-disk failures happen

#32

Post by Dylan16807 »

Ologn wrote: post #20
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

Yes. The decision to use RAID5 instead of RAIDs 1, 10 or 0+1 shows a decision to cut costs. With mirroring, the mirror drives would take over when a disk failed. Also, only one hot spare is in the set. Another cost saving. Yet another decision - RAID5 across 7 disks plus a hot spare. Instead of say across 6 disks or 5 disks. You have two more chances for a disk to go bust and will have to be rebuilt from parity. What…

>With mirroring, the mirror drives would take over when a disk failed.

The mirror drives that also have bad sectors? Then you go even longer without noticing you have problems.

Re: How multi-disk failures happen

#33

Post by X-Istence »

StavrosK wrote: post #25
martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure). The article assumes no scrubbing, which is a stupid thi…

ZFS doesn't do it automatically, you have to crontab it. I have it crontabbed for the 1st or the 15th.

Re: How multi-disk failures happen

#34

Post by mikeash »

papercruncher wrote: post #16
martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I dealt with this exact problem for a number of years. Background scrubbing takes away I/O resources and can be a disaster on your workload if you rely on sequential reads/writes. For that reason, most controllers are configured by default to only scrub when the disk is totally idle which is never. Even if the controller had a better definition of idle, scrubbing an entire disk to find those rotten bits would take a…

With sufficient redundancy available, could you temporarily take a drive out of the RAID for scrubbing, and then add it back in when you're done, to avoid conflicting with ongoing work and destroying linear access patterns?

Re: How multi-disk failures happen

#35

Post by thaumaturgy »

gav wrote: post #21
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

To add some extra emphasis to wazoox's point, RAID-6 is always a better choice than RAID-5. If you're willing take a capacity hit for improved write performance, RAID-1+0 is great. Though you can only survive two disks failing if they are in different pairs. You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations.

> You should also not look at RAID as infallible, if the data is important it should be mirrored in multiple locations.

Right. Because it bears repeating still: RAID is not for backup, RAID is for high availability.

Re: How multi-disk failures happen

#36

Post by corresation »

wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

RAID-5 (or single parity RAID of any kind) is obsolete, period.

RAID-6 offers different compromises relative to RAID-5 (for one, twice the parity space), so it isn't quite like one is the successor of the other. And once you're talking about multiple disk failures, you're at the existential point where you should probably be talking about whole array failures (e.g. your controller has quietly been writing junk for the last hour), and how to deal with that scenario.

Re: How multi-disk failures happen

#37

Post by meaty »

Let us not forget the little considered other cause which usually takes out your tape infrastructure as well: fire.

One outfit I worked decided to stick a brand new APC UPS in the bottom of the rack as it was in their office. It promptly caught fire and burned the entire rack out. The fire protection system did fuck all as well other than scare the shit out of the staff. Scraping molten cat5 cables off with a paint scraper was not fun.

Fortunately it was all covered by the DR procedure. Tip: write one and test it. That's more important than anything.

Re: How multi-disk failures happen

#38

Post by jevinskie »

sschueller wrote: post #4

I have had a multi-disk failure occur with a RAID-1 setup. Server was pre-built from a large vendor and worked fine until both disks failed at the exact same time (within minutes). Took the disks out to find that they had sequential serial numbers. Called vendor for replacement only to have them tell me that they had issues with that batch, yet did not make any attempt to inform me. Spent the day restoring from tape…

I also had a RAID-5 array fail due to 3/4 drives failing near-simultaneously. All of the drives were from the same batch. Some months later, a friend came to me with a computer problem. Her drive had failed. I was able to take a look at the drive and, to my amazement, the drive was from the same batch! Based off of a wild hunch, I swapped the controller board from my one remaining good drive into my friend's drive. The drive worked fine and I was able to recover all her data! It is interesting that the drive failures were likely due to the fact that the drives were from from the same batch but also that fact probably allowed me to seamlessly swap the controller boards!

Re: How multi-disk failures happen

#39

Post by rryan »

StavrosK wrote: post #28
insaneirish wrote: post #3

I'm going to be blunt for a moment. If you are not using ZFS, you deserve what you get. As the author realizes, hardware RAID, or naive software RAID, is becoming more and more useless given the size of volumes and the bit densities (and thus error rates) of those drives. The only solution to this is a proper file system and volume manager that can proactively discover bit rot and give you time to do something about…

How does ZFS discover bit rot if you don't read the bits? I have mine scrubbing once a week, is that unnecessary?

ZFS auto-heals on read so the files you touch regularly are fine. You do need to scrub regularly to prevent bit-rot across all your files. I go with once-a-week as well.

Re: How multi-disk failures happen

#40

Post by VLM »

mikeash wrote: post #34
papercruncher wrote: post #16

Earlier quoted context omitted.

I dealt with this exact problem for a number of years. Background scrubbing takes away I/O resources and can be a disaster on your workload if you rely on sequential reads/writes. For that reason, most controllers are configured by default to only scrub when the disk is totally idle which is never. Even if the controller had a better definition of idle, scrubbing an entire disk to find those rotten bits would take a…

With sufficient redundancy available, could you temporarily take a drive out of the RAID for scrubbing, and then add it back in when you're done, to avoid conflicting with ongoing work and destroying linear access patterns?

The rebuild would be worse than the scrubbing.

A better plan is to light up your disaster recovery plan weekly, and while the DR system is handling the load, scrub to your hearts content on the down system.

Depending on the cost of your hardware vs the cost of your labor vs the cost of downtime, dual servers, one flagged as production and one flagged as development, alternate flags every weekend, might work out. You'll hear lots of bragging about that not being possible because the hardware is too expensive, not so much bragging about labor cost and downtime cost. I worked at financial services corp about two decades ago where downtime was supposedly in excess of $1M/hr. They had triple mainframes set up, basically three machine rooms inside the machine room.

Post reply on HN