Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

231–240 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#231

Post by salawat »

gameshot911 wrote: post #85
salawat wrote: post #68

Somewhat unrelated, but I recently had a motherboard fried by power instability which has given me a healthy respect for the difference between spinning rust and SSD's. My SSD's were scragged. my HDD's were just fine. I guess it's time to figure out how to get a realistic write-through cache setup going, because from now on, if it ain't on spinnin' rust it ain't hard enough yet.

Of what quality was the mobo? I thought the higher-end stuff typically has power protection.

ServalWS B450. Was not at all amused. Oddest bloody thing, because had a pair of Samsung EVO's M.2 NVMes in there, and it still managed to let the magic smoke out.

CPU/GPU and RAM lived, but the corruption of the drives (even if the data was largely recoverable) and rendering of them as inviable to further writes really took me by surprise.

That combined with the way the HDD just did not care one lick despite apparently, just illustrated for me a difference in tolerance to operating conditions that I'd not had the chance to witness first hand yet.

Just figured I'd share while we were talking about SSD weirdness and firmware nonsense.

Re: SSD will fail at 40k power-on hours (2021)

#233

Post by Silhouette »

cyphar wrote: post #163
Hamuko wrote: post #161

Earlier quoted context omitted.

I decided to employ this tactic when I was setting up my new NAS and needed two drives. Upside was that I could definitely know they weren't from the same batch. Downside was that I had to buy a Seagate, and I don't have good experiences with Seagate since my only Seagate drive had died an early death at the tender age of 3. Turns out that this was very much a downside since the Seagate drive died at the tender age o…

I had a series of WD drives that failed, and I managed to get them all replaced under warranty since they died within 3-5 years. I don't buy WD drives anymore but it wasn't the end of the world since I had spares while I waited for the replacement drives to be shipped. Anecdotally, I haven't had issues with Seagate but I'm sure it really boils down to which exact drives you're using and what batch they were in.

I had a series of WD drives that failed, and I managed to get them all replaced under warranty since they died within 3-5 years.

I find hard drive warranties to be mostly an illusion. It's better now that full disk encryption is becoming better supported and potentially available on personal devices and not just corporate ones managed by IT professionals. However until recently the number of drives I've had in any personal/home system that I would have returned under warranty instead of securely destroying to prevent the risk of data leakage was zero. The number of phones I have ever traded in is similarly zero. It's horribly wasteful but until there are cast iron guarantees that all the private data we keep on these devices is going to be securely deleted it's the only sane policy IMHO (apart from never using these devices for anything remotely sensitive in the first place but that's all but impossible in modern society).

Re: SSD will fail at 40k power-on hours (2021)

#234

Post by jstrong »

toast0 wrote: post #97
bayindirh wrote: post #20

Earlier quoted context omitted.

When you purchase a server (fleet), you get a long warranty with it. Generally 3 to 5 years. So you expect this fleet to stay in service for Unless you burn through your SSDs, you're very unlikely to hit this event. When these servers' continue to be used and disks all start to fail at the same time, this will obviously stink. The bathtub curve is not like this. You can feel that.

40k hours is a little more than 4.5 years. These drives deterministically fail at that uptime (unless firmware is updated) and most servers are on 24x7, so if you run your servers for 5 years, it's highly likely you'd run into this. If you run your SSDs hard and they fail early as a result, then you'd be spared from this mass death. Or if you use three year leases and replace on a schedule. Now more than ever, five y…

you have any recs for lower-end bare metal providers?

Re: SSD will fail at 40k power-on hours (2021)

#235

Post by wazoox »

bobsmooth wrote: post #196
mywacaday wrote: post #189

Earlier quoted context omitted.

About 20 years ago I worked for a small storage company, the person that managed the returns of disks form customers was very strongly of the opinion that the odd firmware versions on Seagate drives were returned way more often than the even.

That's the kind of superstition that's only brought about by deep trauma.

back then working with Seagate drives was a trauma in itself. In 2008/2009, I've setup more than 3000 1 TB Barracuda ES drives; 1800 of them failed in the following 3 years (they came with a 5 years warranty). I stopped keeping track of Barracuda failures at some point.

Unsurprisingly, 14 years later I still wouldn't recommend Seagate drives to anyone.

Re: SSD will fail at 40k power-on hours (2021)

#236

Post by ericbarrett »

ekidd wrote: post #229
taneq wrote: post #134

Earlier quoted context omitted.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

I have definitely seen RAID arrays where the drives were all part of a single manufacturing batch, and multiple drives all failed in rapid succession. I think this can be caused by several things: - Unless you periodically do full-drive reads, you may silently accumulate bad blocks across multiple drives in an array. When you finally detect a failed drive, you discover that other drives have also been failing for mon…

You are spot on with everything, especially RAID controllers.

I used to help manage a large fleet of database servers. We found that blocks could "rot" on the underlying storage, yet if they were read often enough they would be held in memory for months and never re-read from the underlying drive. Until you rebooted!

Re: SSD will fail at 40k power-on hours (2021)

#237

Post by cyphar »

Silhouette wrote: post #233
cyphar wrote: post #163

Earlier quoted context omitted.

I had a series of WD drives that failed, and I managed to get them all replaced under warranty since they died within 3-5 years. I don't buy WD drives anymore but it wasn't the end of the world since I had spares while I waited for the replacement drives to be shipped. Anecdotally, I haven't had issues with Seagate but I'm sure it really boils down to which exact drives you're using and what batch they were in.

I had a series of WD drives that failed, and I managed to get them all replaced under warranty since they died within 3-5 years. I find hard drive warranties to be mostly an illusion. It's better now that full disk encryption is becoming better supported and potentially available on personal devices and not just corporate ones managed by IT professionals. However until recently the number of drives I've had in any pe…

All of my drives have FDE. I wouldn't have shipped the drives if that wasn't the case (also luckily the issue was that writes only failed on part of the disk so I could wipe the luks metadata section).

Re: SSD will fail at 40k power-on hours (2021)

#238

Post by no-s »

kqr wrote: post #152
dredmorbius wrote: post #106

Earlier quoted context omitted.

Within an earlier thread "Tell HN: HN Moved from M5 to AWS", there's an excellent comment by loxias about risk diversification across multiple factors. Well worth reading: https://news.ycombinator.com/item?id=32031655 I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable tot…

> I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or ev…

> It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or even increasing it in favour of other properties.

What’s funny is I seem to have to explain this to senior management anew every 6-7 years. I know they teach it in management school, but it in the real world somehow people fall into the false equivalence when they get promoted. Often they adopt a cartoonish view of things because they can’t get the quantitative signals and everything decision effectively reduces to what I ironically term as anecdotal analytics.

I have this amusing heuristic for risk acceptance which I often use to help people approach decisions: you should kick the decision up to someone with higher authority if your signing authority is less than: risk coefficient times quantified exposure, less mitigation cost where mitigation is within signing authority AND/OR budgeted and authorized spend. I like to view mitigation and opportunity cost/benefit in a similar way so I have some idea of equivalences when evaluating tradeoffs.

It’s not original with me, I must have lifted it from some decades-past HBR article or 60’s rant on quantitative business management.

I could rant on various aspects of risk management application all day, thank goodness I've managed to quit before I really got started sharing...in my experience it’s been very helpful when applied in real world engineering implementations.

Re: SSD will fail at 40k power-on hours (2021)

#239

Post by no-s »

dredmorbius wrote: post #106
Silhouette wrote: post #98

Earlier quoted context omitted.

This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.

Within an earlier thread "Tell HN: HN Moved from M5 to AWS", there's an excellent comment by loxias about risk diversification across multiple factors. Well worth reading: https://news.ycombinator.com/item?id=32031655 I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable tot…

> Total system outage is one level, unrecoverable total system outage is even worse.

Ha, I used to suggest people consider “total failure of business” in exposure quantification...

Re: SSD will fail at 40k power-on hours (2021)

#240

Post by user8139471 »

bo0tzz wrote: post #192
zh3 wrote: post #187

Earlier quoted context omitted.

I seem to have the world's oldest SSD (or am I misinterpreting the output?) (shell 1) ~# smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours_and_Msec 0x0032 000 000 000 Old_age Always - 933932h+27m+33.940s

That would make for 106 years of power on time, so it's probably not right...

That is one of the most careful uses of "probably" that I have ever seen.
Post reply on HN