Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My first in-prod corrupted hard drive problem

blog.pavementlink.ch

Post reply on HN
Print view View original

21–30 of 41 posts

Re: My first in-prod corrupted hard drive problem

#21

Post by jeffbee »

proactivesvcs wrote: post #20
jeffbee wrote: post #19

Earlier quoted context omitted.

Wasn't it a conclusion of the Google hard drive reliability study that models based on SMART were not useful? I.e. drives with sector reallocations are much more likely to fail than those without, but their failure rate is still something like 15% per year, so what useful thing can you do with that signal?

Well I don't see why you'd want to keep running a drive that is showing warning signs, it's just asking for trouble. But even if one doesn't replace them from this data, if you start seeing alerts and at the same time your database suffers from corruption, that also shows the use of SMART.

Because taking drives out of service for SMART signals would cost a fortune and almost none of those drives were actually going to fail.

Re: My first in-prod corrupted hard drive problem

#23

Post by estimator7292 »

proactivesvcs wrote: post #20
jeffbee wrote: post #19

Earlier quoted context omitted.

Wasn't it a conclusion of the Google hard drive reliability study that models based on SMART were not useful? I.e. drives with sector reallocations are much more likely to fail than those without, but their failure rate is still something like 15% per year, so what useful thing can you do with that signal?

Well I don't see why you'd want to keep running a drive that is showing warning signs, it's just asking for trouble. But even if one doesn't replace them from this data, if you start seeing alerts and at the same time your database suffers from corruption, that also shows the use of SMART.

N=1, but I had a drive show catastrophic SMART failures once. I figured I'd take the opportunity to tinker with the exposed serial port on the drive's PCB and wiped the SMART values.

Funny thing was, I didn't actually observe any data loss. I stressed the drive for several days, no errors. It went back in my daily driver for the next 5 years with no failure. It's been 15 years since that happened and the drive still hasn't failed.

I don't trust SMART anymore.

Re: My first in-prod corrupted hard drive problem

#24

Post by louwrentius »

> This disk was probably dying. I did some research, and a RAID wouldn’t have saved it either, RAID protects against drive failure, not against silent page corruption that gets faithfully replicated to every mirror.

I dispute this was a 'silent' drive error as many systems reported read errors. Silent data corruption on hard drives is extremely rare, due to the tons of checksums used on all data. Maybe I'm wrong but I bet there are read errors on the drive in the appropriate system logs.

I feel that people confuse regular 'bad blocks' with 'silent data corruption' and there is a huge difference[0].

[0]: https://louwrentius.com/what-home-nas-builders-should-unders...

Re: My first in-prod corrupted hard drive problem

#25

Post by justinclift »

r1chk1t wrote: post #11
pshirshov wrote: post #8

So, you were not using a striped mirror ZFS for a prod database? What could go wrong, yep.

learned the hard way

Yet at the end it still has this:

> I did some research, and a RAID wouldn’t have saved it either, RAID protects against drive failure, not against silent page corruption that gets faithfully replicated to every mirror.

That being said, the article has some strong signal of AI writing in it. So it's possible the author isn't really learning well from the experience either. :(

Re: My first in-prod corrupted hard drive problem

#26

Post by phoronixrly »

louwrentius wrote: post #24

> This disk was probably dying. I did some research, and a RAID wouldn’t have saved it either, RAID protects against drive failure, not against silent page corruption that gets faithfully replicated to every mirror. I dispute this was a 'silent' drive error as many systems reported read errors. Silent data corruption on hard drives is extremely rare, due to the tons of checksums used on all data. Maybe I'm wrong but…

Agreed on the error not being silent. Also incorrect about RAID being unable to catch silent errors - it depends on the implementation - in Linux there's lvmraid that has the option to enforce integrity. There is also zfs which on top of everything else, has RAID functionality and integrity enforcement.

Re: My first in-prod corrupted hard drive problem

#28

Post by jwrallie »

Felger wrote: post #14

Hi, I believe you are quite new to workstation/hardware admin. Lots of things to say here (not native english speaking so basic style, sorry for that) : Disk errors logged in the system event log are from the I/O layer, low-level class driver (msahci.sys) / filter drivers. See Windows Storage Driver Architecture : https://learn.microsoft.com/en-us/windows-hardware/drivers/s... A disk error of this type showing in the…

If taking it offline is not a concern, I would try a low level backup with ddrescue while booting from external media as soon as possible.

Keep using the system from a disk showing read issues could trigger loss of more data, and one could always back up the SQL from the backup image later.

Re: My first in-prod corrupted hard drive problem

#29

Post by pshirshov »

justinclift wrote: post #25
r1chk1t wrote: post #11

Earlier quoted context omitted.

learned the hard way

Yet at the end it still has this: > I did some research, and a RAID wouldn’t have saved it either, RAID protects against drive failure, not against silent page corruption that gets faithfully replicated to every mirror. That being said, the article has some strong signal of AI writing in it. So it's possible the author isn't really learning well from the experience either. :(

ZFS and ECC do protect against silent page corruption that gets faithfully replicated to every mirror.
Post reply on HN