Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

211–220 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#211

Earlier quoted context omitted.

About 20 years ago I worked for a small storage company, the person that managed the returns of disks form customers was very strongly of the opinion that the odd firmware versions on Seagate drives were returned way more often than the even.

That's the kind of superstition that's only brought about by deep trauma.

Thinking back mixing firmware versions on new units was avoided where possible and we would also try to replace like for like firmware on RMAs

Re: SSD will fail at 40k power-on hours (2021)

#212
post #9
post #7

Earlier quoted context omitted.

Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…

the chance of two SSD's failing at the same time under normal circumstances is extremely slim. So this might actually be a good cause of this incident.

True. But it's more about the probability of things being "normal", isn't it? I had multiple Evo970 fail within a very short time. Turns out to be a systematic problem of drives produced in one specific month.

Just how much difference is enough to be safe is the price question...

Re: SSD will fail at 40k power-on hours (2021)

#213
post #194

Earlier quoted context omitted.

Okay, fair enough. To me, the risk involvement is so obvious (to any serious business function!) that I find the management/reduction distinction a more important point. But I can see it your way too.

Thanks. I don't know how long you've been in the business, but the change seems a relatively recent one, one that wasn't manifestly obvious to me, and one that has pretty much always seemed difficult to communicate to management. Whether that's because business management is often about ignoring risks or treating it as inconvenient, or if I've just had a long string of bad bosses, I'm not sure. I did make a point of…

You know what? You're right again!

The stuff I've read that touches on this idea is almost all from 2006 and onwards, mainly 2010s. The earliest example is a bit of an outlier: Douglas Hubbard's 1985 How to Measure Anything -- but it's also only tangentially related.

The other real exceptions are books on statistics (where the idea of risk management -- at least in my collection -- seems to have gotten popular in the 1950s, probably as a result of the second World war) and financial risk management (which seems to really have taken off in the 1980s, probably in conjunction with options becoming a thing.) Statisticians and finance people (and by extension e.g. poker and bridge players) have known this stuff for a while.

Of course, hydrologists have been doing this stuff since the early 1900s at least, but extreme value theory has always been a kind of niche so I'm not sure I should count that.

----

That said, I did mention it was obvious to me. I still find it hard to convince management and colleagues of its importance...

Re: SSD will fail at 40k power-on hours (2021)

#214

4 byte integer rollover: 2^32=4,294,967,296/4.5=954,437,177/365=2,614,896/3600=726/24=30ticks/sec

Unlikely; a clock running at 30 Hz overflows a 32 bit (unsigned) integer in 39,768 hours, while the HN disk failed after at least 39,984 hours [1], and the vendors wouldn't issues warnings about 40,000 hours if it actually fails about 230 hours before that. [1] https://news.ycombinator.com/item?id=32031428

Not sure about the mfgr- could be rounding for memory saliency. . . but re HN they’re counting clock time not power on time. Could have easily been turned off total 10 days for maintenance etc. (and good routine could have both servers off about the same no hrs.) That’s only 0.6% downtime.

Re: SSD will fail at 40k power-on hours (2021)

#216
post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

How to do this on a Windows PC?

Thanks

Re: SSD will fail at 40k power-on hours (2021)

#218
post #202
post #163

Earlier quoted context omitted.

I had a series of WD drives that failed, and I managed to get them all replaced under warranty since they died within 3-5 years. I don't buy WD drives anymore but it wasn't the end of the world since I had spares while I waited for the replacement drives to be shipped. Anecdotally, I haven't had issues with Seagate but I'm sure it really boils down to which exact drives you're using and what batch they were in.

Pick a manufacturer and you'll be able to find plenty of horror stories. Some may be worse than others but diversification is the right answer anyway.

But how do you color match your drives in your spiffy NAS?

This is why my bicycle drivetrain should be a frankenstein combination of parts from different manufacturers?

/s

Re: SSD will fail at 40k power-on hours (2021)

#219
post #73

Earlier quoted context omitted.

Just FYI for anyone for whom this didn't work by default: I needed to use the --all flag with smartctl (and install smartmontools if you don't have it).

The -a flag from my example should be an alias for --all (man smartctl | grep -A1 ' -a' | head -2). Is that not the case in your version?

Pro tip: when writing out commands for people to read it helps to use the long form arguments. In this case passing '--all' instead of '-a' to smartctl. It makes it easier to read and more clear what specific options do. Same with calling things in scripts. Short form is for quick and dirty typing things, but not great for reading or comprehension :)

Re: SSD will fail at 40k power-on hours (2021)

#220
post #204
post #127

Earlier quoted context omitted.

We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently…

The deathstars were fantastic, they almost always failed on the outer edges of the platters. So if you only formatted them (filesystem wise) out to capacity-2Gb they were a really cheap option at the time.

That’s a very different definition of fantastic than the one I use.
Post reply on HN