Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

111–120 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#111

Post by pclmulqdq »

walterbell wrote: post #89

Would companies be willing to contribute to the OpenSSD project? OCP (Open Compute Project) has shown that customer-operators can cooperate on open hardware designs, successfully influencing enterprise hardware supply chains. Commercial DPUs and SmartNICs were preceded by a decade of open hardware and research by the NetFPGA project ( https://netfpga.org ). Why not DiskFPGA? 2017 OpenSSD overview, based on Xilinx: ht…

An FPGA-based SSD will always be too expensive for people with truly large scale. The controller ASICs are a lot cheaper.

Re: SSD will fail at 40k power-on hours (2021)

#112

Post by kristopolous »

m0llusk wrote: post #35
pmoriarty wrote: post #27

Earlier quoted context omitted.

One thing everyone could and should be doing is backups.

Two things: Test restores or you don't actually have backups. Just saying.

Depending on your usecase you can integrate using your backups occasionally into your normal data processing.

Again, depends on usecase but then it becomes integral to your existing workflow instead of an addendum that you end up forgetting to do

The whole purpose is to make the failing of one be effectively extremely noisy and irritating

It's like what I do with raid. I have a script that will shut the machine down on drive failure and then will use dialog(1) to say something like "hey bozo replace the fucking drive first" when you boot it up and then it will shutdown again and be unusable.

Make the complaining show stopping, loud, rude, and disruptive. Because if the next one fails you're screwed

Re: SSD will fail at 40k power-on hours (2021)

#113

Post by Silhouette »

colejohnson66 wrote: post #108
Silhouette wrote: post #98

Earlier quoted context omitted.

This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.

If you’re willing to wait, you can always order half, wait a month or two, then order the other half.

Unfortunately this particular server was a replacement for another that had failed suddenly so that wasn't really an option in my case. If it had been one of many at work then it would have been a sensible option, I agree.

Re: SSD will fail at 40k power-on hours (2021)

#114

Post by ratsmack »

belkarx wrote: post #110
colejohnson66 wrote: post #109

Earlier quoted context omitted.

An open source SSD is also a lot more feasible than an open source hard drive. Even if you managed to get an open source HDD controller, you still need the precision mechanical parts that are impossible for the average person to make. With SSDs, however, it’s just a PCB with ICs. Edit: this obviously ignores any troubles one would have sourcing the ICs (such as possible NDAs)

Is there anything special about making SSDs that the average person would not be able to do or is it a "if you can outsource PCB printing and maybe solder you can make one" situation?

The limiting factor would be the memory chips themselves and any firmware required for them (if any). I also don't know how well they are spec'd and if full documentation is available without NDA's and lawyers.

Re: SSD will fail at 40k power-on hours (2021)

#115

Post by yalogin »

What exactly is causing the bug though? If the same area is written to some x number of times the solid state device in that location permanently fails, is that correct? If so in an always on device how can this bug be escaped? They need to randomize the writes and for that the storage size should be a multiple of what is needed for regular operation. Even then the disk will fail eventually. What am I missing here?

Re: SSD will fail at 40k power-on hours (2021)

#116

Post by dylan604 »

freebreakfast wrote: post #82
lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

iMac mid-2010. Original disk drive. "9 Power_On_Hours 0x0032 001 001 000 Old_age Always - 74233" 12 years old. More than 8 years of run time. It keeps on purring. Yes, I have redundant backups. I also have a replacement drive ready. I just want to see how far I can take it.

Luckily it is old enough where you can replace it

Re: SSD will fail at 40k power-on hours (2021)

#117

Post by bell-cot »

A Modest Proposal: All $LARGE business insurance policies specify that, to the extent to which any insured loss was caused or worstened by reliance upon the correct functioning of SSD or related drive technologies...YOYO, and any & all losses are solely on you.

Re: SSD will fail at 40k power-on hours (2021)

#118

Post by rexf »

chrischen wrote: post #61
m0llusk wrote: post #35

Earlier quoted context omitted.

Two things: Test restores or you don't actually have backups. Just saying.

I got bit by this with iPhone backups. I did a phone trade in and followed the backup before trading in instructions. Problem is after the trade in the backup failed to restore due to an unknown error. The whole manual syncing and backing up with a cable workflow with Apple is super fickle and riddled with bugs. Luckily I had Time Machine backups of my iOS backups and I managed to avoid losing too much data. As a sid…

While I don't like how annoying Apple is with service upselling (iCloud, Music, Arcade), at least they moved iPhone backup from iTunes to Finder. So their local iPhone backup process is being maintained over time.

I don't have issues with my computer (PC or Mac) detecting my iPhone. Generally need to make sure iPhone is unlocked after plugging it in. What is tough is the large size of my iPhone (X gb) and how small my Mac's HD is (2X gb).

Re: SSD will fail at 40k power-on hours (2021)

#119

Post by fsckboy »

yalogin wrote: post #115

What exactly is causing the bug though? If the same area is written to some x number of times the solid state device in that location permanently fails, is that correct? If so in an always on device how can this bug be escaped? They need to randomize the writes and for that the storage size should be a multiple of what is needed for regular operation. Even then the disk will fail eventually. What am I missing here?

there is a bug in a software counter overflowing which a firmware upgrade will fix, is what it says.

Re: SSD will fail at 40k power-on hours (2021)

#120

Post by wyager »

solardev wrote: post #7
dredmorbius wrote: post #2

Possibly related to recent HN issues, see: https://news.ycombinator.com/item?id=32031243

Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…

I generally replace HDDs in my personal zpool a few days apart, for this reason. I also order them from different suppliers, so I can get different manufacture dates.
Post reply on HN