Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How multi-disk failures happen

sysadmin1138.net

Post reply on HN
Print view View original

71–80 of 92 posts

Re: How multi-disk failures happen

#71

Post by olgeni »

StavrosK wrote: post #25
martinced wrote: post #7

I'm not a Un x sysadmin at all and don't know much about hard drive: I'm just a software dev. But from the beginning of TFA, after reading this: "Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected." The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting s…

I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure). The article assumes no scrubbing, which is a stupid thi…

You may add something like this to /etc/periodic.conf:

daily_status_zfs_enable="YES" daily_scrub_zfs_enable="YES" daily_scrub_zfs_default_threshold="6" # in days

and it will scrub the pools every 6 days (and send you a report in the daily run output).

Re: How multi-disk failures happen

#72

Post by wazoox »

corresation wrote: post #36
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

RAID-5 (or single parity RAID of any kind) is obsolete, period. RAID-6 offers different compromises relative to RAID-5 (for one, twice the parity space), so it isn't quite like one is the successor of the other. And once you're talking about multiple disk failures, you're at the existential point where you should probably be talking about whole array failures (e.g. your controller has quietly been writing junk for th…

>it isn't quite like one is the successor of the other.

Given the current price of hard drives, I don't get how "twice the parity space" can even matter. Furthermore, modern RAID controllers perform almost exactly the same using RAID-5 or RAID-6 (verified on most 3Ware, LSI, Adaptec and Areca controllers).

So yes, RAID-6 definitely is RAID-5 successor.

> how to deal with that scenario.

RAID is not an alternative to backup and never was. You deal with that scenario through proper backup or replication.

Re: How multi-disk failures happen

#73

Post by meaty »

jzwinck wrote: post #70
meaty wrote: post #37

Let us not forget the little considered other cause which usually takes out your tape infrastructure as well: fire. One outfit I worked decided to stick a brand new APC UPS in the bottom of the rack as it was in their office. It promptly caught fire and burned the entire rack out. The fire protection system did fuck all as well other than scare the shit out of the staff. Scraping molten cat5 cables off with a paint s…

Do you know what caused the fire? Improper installation? Overheating? Defective equipment?

Defective APC charge regulator apparently. To be honest they were good and their insurance paid out very quickly to both us and the site owners.

Re: How multi-disk failures happen

#74

Post by wazoox »

caw wrote: post #44
wazoox wrote: post #5

Using RAID-5 is the primary error here. RAID-5 (or single parity RAID of any kind) is obsolete , period. The story here doesn't ring true to me to be honest; I'm currently herding several hundred multi-terabytes servers, and multiple drive failures appear in one and only case: when using Seagate Barracuda ES2 1TB or WD desktop class drives. These are the two very problematic setups. In all other cases, use RAID-6 and…

EMC uses RAID5 as the default for storage arrays, and then has some number of global hot spares. Netapp uses RAID6 by default, and then also has some number of hot spares. I've never had data loss from either system as a result of multi-drive failure. RAID5 is perfectly fine in most instances. Desktop drives will drop out of RAID arrays frequently, so you have to use RAID6 if you choose to go that route. If a disk dr…

> RAID5 is perfectly fine in most instances.

If you're using low capacity, enterprise class SAS drives, mostly, yes. However when using large capacity SATA drives, it most definitely isn't.

SATA drives (even "enterprise" SATA drives) have an official unrecoverable error rate of 1/10^14. From my experience, the truth is more like 1/10^13.

10^13 bits is roughly 10 terabytes. That means that every time you're reading 10 TB, you are statistically certain to encounter an unrecoverable bit error (and have a 1% chance of having 100 errors, of course). In the case of a rebuilding 10 TB array (only a couple 3 or 4 TB drives) using RAID-5, that means that you're almost sure to have an ECC error that will prevent you from ever rebuilding properly without corruption.

Re: How multi-disk failures happen

#75

Post by wazoox »

ars wrote: post #63
andrewcooke wrote: post #47

Earlier quoted context omitted.

so how is that better than raid? it sounds identical to what is described in the article - the problem was not scrubbing (afaict).

ZFS is at the file level, not the disk level. So recovering from errors (i.e. rebuilds) is MUCH faster.

Only when the array is mostly empty. For several years hardware RAID controllers have been rebuilding only used space, too. Really, ZFS isn't that much of the miracle some want it to be.

Re: How multi-disk failures happen

#76

Post by CrLf »

One thing I've learnt early on my career as a sysadmin is that disk quality is very important, and so is the quality of the RAID controller or software RAID subsystem. After you have a multiple drive failure on a supposedly safe RAID-1, and get forced into stitching it back into operation with a combination of "badblocks" and "dd", you'll quickly understand why...

A good RAID controller won't let a drive with bad sectors continue to operate silently in an array. Once an unreadable sector is detected, the drive is failed immediately, period.

The problem is in the detection, but good RAID controllers "scrub" the whole array periodically. If they don't, or if you are paranoid like me, the same can be accomplished by having "smartd" initiate a long "SMART" self-test on the drives every week.

Good controllers will even fail drives when a transient error happens, one which triggers a bad block reallocation by the drive, for example. This is what makes some people fix failed drives by taking them out and putting them back in. After a rebuild the drive will operate normally without any errors, but you are putting yourself at a serious risk of it failing during a rebuild if another drive happens to fail, so DON'T do this.

Some others will react differently to these transient errors. EMC arrays, for instance, will copy a suspicious drive to the hot-spare and call home for a replacement. This is much faster than a full rebuild, but also much safer because it doesn't increase the risk of a second drive failing while doing it.

Oh, and did I mention that cheap drives lie?

Avoid using desktop drives on production servers for important data, even in a RAID configuration, if you don't have some kind of replicated storage layer above your RAID layer (meaning you can afford recovering one node from backup for speed and resync with the master to make it current).

Re: How multi-disk failures happen

#77

Post by CrLf »

akg_67 wrote: post #67
VLM wrote: post #41

My favorite raid 5 failure mode is when a old hardware card fails and you have no stock spare and the maint contract was not renewed years ago and the cost of a new card is too much to be expensed and buying as a capital replacement will take a week or two minimum to be approved. And/or the card has been discontinued so you have to buy a used one from a shady foreign surplus dealer. I've seen too much of this type of…

You don't need to replace old/discontinued RAID card with the used one (assuming you mean same make and model). The new card from same manufacturer will work. The RAID card writes DCBs on disks that are read by the new RAID card upon replacement.

"The RAID card writes DCBs on disks that are read by the new RAID card upon replacement."

Which also makes for very nice RAID failures, like this one that has happened to me on an HP controller:

A drive fails because of some SCSI electronics problem and when you replace it, the controller gives it a different SCSI ID. Now, the controller maps RAID arrays to drives, and it is now impossible to add the replacement drive to the degraded array because SCSI IDs in these controllers aren't user defined and the controller doesn't allow the degraded array to be modified.

And since the controller has now happily written it's configuration onto the drives, it doesn't matter that you shuffle the drives around to try to force the controller into giving up it's internal configuration.

Oh, and the controller is an onboard controller, so you can't just replace it with another one (which would also read the configuration on disk and put himself in the same stupid state, I suppose).

Re: How multi-disk failures happen

#78

Post by StavrosK »

olgeni wrote: post #71
StavrosK wrote: post #25

Earlier quoted context omitted.

I have a home server with three disks and ZFS, for my photos and things, so I'm not an expert. However, Ubuntu's md-raid includes scrubbing once a week by default, and I added scrubbing to my ZFS setup via crontab, again once a week (I'm not sure if ZFS does it automatically, but I don't think it does. I would appreciate a correction, if someone knows for sure). The article assumes no scrubbing, which is a stupid thi…

You may add something like this to /etc/periodic.conf: daily_status_zfs_enable="YES" daily_scrub_zfs_enable="YES" daily_scrub_zfs_default_threshold="6" # in days and it will scrub the pools every 6 days (and send you a report in the daily run output).

Very nice, thank you, I will try that. I am rather dismayed, however, by learning in this thread that my disks have 4K sector sizes and ZFS autodetected 512 bytes, which means I'll have to destroy the pool and recreate it...

Re: How multi-disk failures happen

#79

Post by olgeni »

StavrosK wrote: post #78
olgeni wrote: post #71

Earlier quoted context omitted.

You may add something like this to /etc/periodic.conf: daily_status_zfs_enable="YES" daily_scrub_zfs_enable="YES" daily_scrub_zfs_default_threshold="6" # in days and it will scrub the pools every 6 days (and send you a report in the daily run output).

Very nice, thank you, I will try that. I am rather dismayed, however, by learning in this thread that my disks have 4K sector sizes and ZFS autodetected 512 bytes, which means I'll have to destroy the pool and recreate it...

It happens all the time...

If you run "camcontrol identify ada0" (or whatever your device is) you can find out before it is too late:

sector size logical 512, physical 512, offset 0

This is from a lucky drive of course :)

Re: How multi-disk failures happen

#80

Post by olgeni »

wazoox wrote: post #75
ars wrote: post #63

Earlier quoted context omitted.

ZFS is at the file level, not the disk level. So recovering from errors (i.e. rebuilds) is MUCH faster.

Only when the array is mostly empty. For several years hardware RAID controllers have been rebuilding only used space, too. Really, ZFS isn't that much of the miracle some want it to be.

How do they know what space is used or unused?
Post reply on HN