Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

131–140 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#131

Post by cmeacham98 »

dtjb wrote: post #34
alliao wrote: post #17

Earlier quoted context omitted.

planned obsolescence is quite a thing...?

In some cases, but a product must fulfill its core purpose. If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand. Same way a car doesn't adopt planned obsolescence by blowing up after 200k miles.

> If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand.

Other than "intentionally" (which we cannot know and makes no difference to whether you lose your data or not) that is literally what these SSDs are doing, and no SSD brand has been destroyed over it.

Re: SSD will fail at 40k power-on hours (2021)

#132

Post by dredmorbius »

Silhouette wrote: post #113
colejohnson66 wrote: post #108

Earlier quoted context omitted.

If you’re willing to wait, you can always order half, wait a month or two, then order the other half.

Unfortunately this particular server was a replacement for another that had failed suddenly so that wasn't really an option in my case. If it had been one of many at work then it would have been a sensible option, I agree.

Repurposing several of the initial drives with later purchases (or exchanges) might be another option.

Re: SSD will fail at 40k power-on hours (2021)

#133

Post by dredmorbius »

oogali wrote: post #127
Group_B wrote: post #86

Earlier quoted context omitted.

If both SSD's are from the same lot number and one fails, the chances of the second failing go up by a high amount. Both failing at the same time though is extremely rare.

We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently…

Surprisingly few HN submissions under either term:

IBM Deskstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

IBM Deathstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Re: SSD will fail at 40k power-on hours (2021)

#134

Post by taneq »

mkl wrote: post #11
solardev wrote: post #7

Earlier quoted context omitted.

Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…

Not two SSDs, four : two in the main server, and two in the backup server.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

Re: SSD will fail at 40k power-on hours (2021)

#135

Post by xyzzy_plugh »

dredmorbius wrote: post #133
oogali wrote: post #127

Earlier quoted context omitted.

We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently…

Surprisingly few HN submissions under either term: IBM Deskstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... IBM Deathstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Why would there be submissions? These drives predate HN.

Re: SSD will fail at 40k power-on hours (2021)

#136

Post by yalogin »

fsckboy wrote: post #119
yalogin wrote: post #115

What exactly is causing the bug though? If the same area is written to some x number of times the solid state device in that location permanently fails, is that correct? If so in an always on device how can this bug be escaped? They need to randomize the writes and for that the storage size should be a multiple of what is needed for regular operation. Even then the disk will fail eventually. What am I missing here?

there is a bug in a software counter overflowing which a firmware upgrade will fix, is what it says.

Ah cool, thanks

Re: SSD will fail at 40k power-on hours (2021)

#137

Post by rootw0rm »

jeffbee wrote: post #74
lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

Checked my Samsung 970 Evo 2TB and it says 487 even though it’s been on continuously for years.

My 960 Pro 1TB, which I paid way too much for back in the day is at just shy of 5,000 hours, and I've used the hell out of it since it was new.

Re: SSD will fail at 40k power-on hours (2021)

#138

Post by champtar »

HPE had the same issue on some of their SSDs. We received an advisory months before it would have been a problem, and had time to upgrade all our customers ... except 2 servers that we missed. Luckily when I saw all disk in error in the iLO on one of the server I remembered this issue, googled the model, confirmed it was affected, and was able to shutdown the second server and start the upgrade. Not sure client was happy but at least it was only a 1h outages instead of maybe a day to get the disks + restoring from backup. HPE did replace the disks under warranty.

Re: SSD will fail at 40k power-on hours (2021)

#139

Post by cyphar »

Silhouette wrote: post #98
dredmorbius wrote: post #87

Earlier quoted context omitted.

It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.

This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.

My solution is to use a different manufacturer for each drive in a mirror. The prices are usually pretty similar and you get to make sure that one firmware bug doesn't kill your entire pool.

Re: SSD will fail at 40k power-on hours (2021)

#140

Post by anshumankmr »

lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

How to do this on a Windows PC?
Post reply on HN