Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

171–180 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#171

Post by gfrff »

kqr wrote: post #152
dredmorbius wrote: post #106

Earlier quoted context omitted.

Within an earlier thread "Tell HN: HN Moved from M5 to AWS", there's an excellent comment by loxias about risk diversification across multiple factors. Well worth reading: https://news.ycombinator.com/item?id=32031655 I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable tot…

> I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or ev…

One of the advantages startups have is higher risk tolerance than branded megacorporations. Data, customers, brand, employees, law suits, or cynically, human lives.

Re: SSD will fail at 40k power-on hours (2021)

#172

Post by vidarh »

dredmorbius wrote: post #133
oogali wrote: post #127

Earlier quoted context omitted.

We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently…

Surprisingly few HN submissions under either term: IBM Deskstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... IBM Deathstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

It does however come up in comments regularly. I know I've brought it up more than once, because I had a week long ordeal replacing all the drives in an array as they died one by one back in the day.

Re: SSD will fail at 40k power-on hours (2021)

#174

Post by zymhan »

ghostly_s wrote: post #160

Yikes. Cisco claims this is "an industry wide firmware index bug". Is there any validity to this claim? Are any consumer drives affected by the same issue?

This apparently also happened two years ago?

https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...

Re: SSD will fail at 40k power-on hours (2021)

#176

Post by loxias »

cyphar wrote: post #139
Silhouette wrote: post #98

Earlier quoted context omitted.

This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.

My solution is to use a different manufacturer for each drive in a mirror. The prices are usually pretty similar and you get to make sure that one firmware bug doesn't kill your entire pool.

This is the way.

For even more peace of mind, (and only when you can afford it, obviously) try decoupling your disk purchases a bit from when you're going to need them.

When you see a good price or a sale on a particular disk, grab it add it to your own personal "prebought disk pool". When it's time to either replace a disk or spin up a whole new array, now you have the benefit of diversification across time.

Re: SSD will fail at 40k power-on hours (2021)

#178

Post by vectorcrumb »

nodja wrote: post #142
anshumankmr wrote: post #140

Earlier quoted context omitted.

How to do this on a Windows PC?

In an elevated powershell prompt Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object PowerOnHours

I'm getting a blank value back as well, even with admin privileges.

Re: SSD will fail at 40k power-on hours (2021)

#179

Post by underscore_ku »

anshumankmr wrote: post #140
lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

How to do this on a Windows PC?

install ubuntu linux first

Re: SSD will fail at 40k power-on hours (2021)

#180

Post by mcv »

dredmorbius wrote: post #87
kazen44 wrote: post #9

Earlier quoted context omitted.

the chance of two SSD's failing at the same time under normal circumstances is extremely slim. So this might actually be a good cause of this incident.

It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.

If it's really 4 drives, bought at the same time, failing simultaneously, that's pretty damning evidence.

Remind me of my laptop that I bought with 2 SSDs. Not from HP or Dell, but still, I now wonder if I should replace one of them with a more recent SSD and give the other to my son (he currently has an anemic SSD that's too small to install Genshin Impact on).

Post reply on HN