Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My first in-prod corrupted hard drive problem

blog.pavementlink.ch

Post reply on HN
Print view View original

31–40 of 41 posts

Re: My first in-prod corrupted hard drive problem

#31

Post by justinclift »

pshirshov wrote: post #29
justinclift wrote: post #25

Earlier quoted context omitted.

Yet at the end it still has this: > I did some research, and a RAID wouldn’t have saved it either, RAID protects against drive failure, not against silent page corruption that gets faithfully replicated to every mirror. That being said, the article has some strong signal of AI writing in it. So it's possible the author isn't really learning well from the experience either. :(

ZFS and ECC do protect against silent page corruption that gets faithfully replicated to every mirror.

Yeah, that was my point. The author seems to have gotten things wrong in some fundamental way.

Re: My first in-prod corrupted hard drive problem

#33

Post by rkagerer »

proactivesvcs wrote: post #13

I'm surprised to have read to the end and found that they're still not performing any hardware monitoring and alerting. SMART may not always show up pre-failure warnings but when it does they can usually be trusted.

Hard Disk Sentinel is really good for this type of thing. The developer is awesome and some years ago after I asked for some new features added code to better support my RAID adapter.

Re: My first in-prod corrupted hard drive problem

#34

Post by adastra22 »

ryandrake wrote: post #9
toast0 wrote: post #7

Earlier quoted context omitted.

> Over the last decade, I've ran hundreds of servers if not thousands, and I entirely stopped using hard drives, now it's solely SSD/NVMe where the failure rate in practice is incredibly lower, I've had my fair share of middle-night runs because websites are offline or whatever to end-up in a hard drive diagnosis circus. My experience is that (most) spinners give off reliable pre-failure indicators (if you take the t…

> My experience is that (most) spinners give off reliable pre-failure indicators (if you take the time to look/script looking), but SSDs fail by disappearing from the bus. The SSDs do fail much less often, but they still fail from time to time and recovery is harder. I'm not a pro, just a smalltime dork with a homelab. I use cheap WD HDDs on my NAS system connected to an LSI hardware RAID controller. I'll boast that…

I have a homelab with 24 disks (2x 12-drive raidz3 pools). In the past 15 years of operation I have replaced many drives. Usually my experience matches yours, but there is one time I got a simultaneous 3-disk failure. One disk failed, then when I replaced it and did a scrub, two more from the same pool failed in quick succession. I had to scramble to get more spare drives, and didn't sleep much while I waited for the pools to finish the scrub.

Coordinated failure is a thing, unfortunately. Drives bought together tend to fail together.

Re: My first in-prod corrupted hard drive problem

#35

Post by proactivesvcs »

estimator7292 wrote: post #23
proactivesvcs wrote: post #20

Earlier quoted context omitted.

Well I don't see why you'd want to keep running a drive that is showing warning signs, it's just asking for trouble. But even if one doesn't replace them from this data, if you start seeing alerts and at the same time your database suffers from corruption, that also shows the use of SMART.

N=1, but I had a drive show catastrophic SMART failures once. I figured I'd take the opportunity to tinker with the exposed serial port on the drive's PCB and wiped the SMART values. Funny thing was, I didn't actually observe any data loss. I stressed the drive for several days, no errors. It went back in my daily driver for the next 5 years with no failure. It's been 15 years since that happened and the drive still…

I've known a fair few drives with uncorrectable sectors or adaptor warnings that continue to work but a lot more that have degraded or just outright failed.

Re: My first in-prod corrupted hard drive problem

#36

Post by sitzkrieg »

pshirshov wrote: post #8

So, you were not using a striped mirror ZFS for a prod database? What could go wrong, yep.

the article describes why they can’t use zfs due to windows server requirements. this is the right choice because zfs on windows is completely garbage

Re: My first in-prod corrupted hard drive problem

#37

Post by prirun »

My sister has a Windows 10 laptop she used for her accounting business. One day it decided not to boot, saying there was no boot device. I took the laptop home, took the SSD out (Samsung 1TB), put it in an external USB case, plugged it into another Windows laptop, and it showed up in Explorer. Weird.

I had another brand-new, identical Samsung SSD, so I hooked both the old and new drive up to a Linux laptop (with USB cases) and tried to dd the old drive to the new drive. That mostly worked, but VERY VERY slowly: it would run fast for 5 seconds and then have no activity for 30 seconds. I had a fan blowing on the old drive to keep it cool because it was running very hot.

The dd copy would eventually fail and then I'd restart it with appropriate iseek and oseek values. I also did a cmp /dev/zero with the new disk to verify that it was all zeroes (it was brand new), and that allowed me to use conv=sparse on the dd. The reason for that was to avoid writing to ever sector of the new disk; I didn't want to copy sectors from the old drive that had never been accessed (she only used about 250GB of the 1TB).

It took a couple of days and about 5 restarts to finish the copy, but it did work, and as a precaution, I made another copy of the drive and ran a cmp of the original drive and the 2nd copy (also having to restart cmp several times). Since that compare worked, I knew that all 3 drives had identical content. The new drive worked fine in her laptop and she was mighty glad to see her Windows login screen.

The thing that made this work, IMO, is that Linux has a longer timeout for errors than Windows apparently does, especially during the boot sequence. Plus Linux allows adjusting the drive timeout, so if the device is doing error recover, which is sometimes slow, it gives it time to finish rather than reporting an error.

One of my theories was that the bad SSD was overheating, but if that was the case, a cold boot should have worked, with the failures only coming later.

The other theory is that one of the chips on the SSD failed, so the drive was having to use the ECC codes to correct for the missing information, and the correction process was taking longer than Windows boot would tolerate.

Re: My first in-prod corrupted hard drive problem

#38

Post by alternatetwo »

jtchang wrote: post #3

Confused as to the actual root cause. Don't all hard drives provide SMART diagnostics these days? Was it really bad sectors?

I’ve had a disk silently corrupt data while reading it without SMART data showing any reallocated sectors at first. at some point it did and Seagate was able to recover many files from it.

Re: My first in-prod corrupted hard drive problem

#39

Post by BenjiWiebe »

prirun wrote: post #37

My sister has a Windows 10 laptop she used for her accounting business. One day it decided not to boot, saying there was no boot device. I took the laptop home, took the SSD out (Samsung 1TB), put it in an external USB case, plugged it into another Windows laptop, and it showed up in Explorer. Weird. I had another brand-new, identical Samsung SSD, so I hooked both the old and new drive up to a Linux laptop (with USB…

Next time you have a disk where you need to do repeated dd runs over different ranges, or suspect that you might need to, use ddrescue. It tracks which sectors have been recovered (and has lots of useful options).

You can also get 'partclone' to generate a list (in ddrescue format) of sectors containing data, so you don't need to try to read unused areas of the disk. For the partclone trick to work, the FS does need to be at least somewhat readable.

Re: My first in-prod corrupted hard drive problem

#40

Post by evanjrowley »

Since the author is using Windows, I'm surprised nobody has recommended Microsoft's Resilient File System (ReFS). It's kind of like Microsoft's answer to ZFS, but with support for clustered filesystems. Not a bad choice here if you're using the Microsoft stack: https://learn.microsoft.com/en-us/windows-server/storage/ref...
Post reply on HN