Backblaze have a great blog about things they learn about hard drives. It's been going for years, less about firmware issues and more about general usage. https://www.backblaze.com/blog/backblaze-drive-stats-for-q1-...
SSD will fail at 40k power-on hours (2021)
21–30 of 276 posts
Re: SSD will fail at 40k power-on hours (2021)
#22Possibly related to recent HN issues, see: https://news.ycombinator.com/item?id=32031243
Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…
[deleted]
Re: SSD will fail at 40k power-on hours (2021)
#23Earlier quoted context omitted.
Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you s…
Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?
If a spinning rust can run for ~8 years without any problems,a consumer SSD can hit beyond 40K hours reliably, and everything is checked and tested tens of times because of the complexity of flash storage, I'd get suspicious too.
Also, enterprise drives get firmware updates (regardless of spinning or not), and this firmware is automatically applied via RAID controller, so it could be remedied easily before it got this big if it's an actual error.
Re: SSD will fail at 40k power-on hours (2021)
#24crap so its certainly HP laptops. so which laptops are safe from this?
My HP laptop has Toshiba SSD. I'm not sure about other models. But I think only enterprise SSDs are affected.
Re: SSD will fail at 40k power-on hours (2021)
#25Earlier quoted context omitted.
Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…
Not two SSDs, four : two in the main server, and two in the backup server.
Thanks for the correction!
Re: SSD will fail at 40k power-on hours (2021)
#26Earlier quoted context omitted.
Someone pointed out on the other thread that it could be 2^57 nanoseconds: >>> 2**57/10**9/3600 40031.996687737745
If it were 53, I'd wonder "are they storing the time in the integer part of a double precision float?" That wouldn't go negative, it'd just start absorbing increments without changing the value. Though that might cause a divide by zero? What could cause unexpected behavior at 57 bits? Perhaps storing fractions of an hour, like incrementing it every 1/16th of an hour and calculating a relative rate of change, causing…
My overactive imagination thinks it went something like this:
Engineer A: Gee, I need to store a few flags with each block, but there's nowhere to put them. Ah! We're storing timestamps as 64-bit microseconds. I can borrow a few of those bits and there'll still be enough to go for thousands of years without overflowing.
Engineer B: Gee, our SSDs are getting so fast, soon we'll be able to hit 1M writes/sec. But we're storing timestamps as microseconds. How can we generate unique timestamps for each write? Ah! I'll switch to nanoseconds. It's a good thing we have plenty of space in this 64-bit int.
BOOM!
Re: SSD will fail at 40k power-on hours (2021)
#27It's been over two years since this was first identified... since this apparently affected many makes and models of SSDs, it would be nice to know if my laptop could be affected and if there's anything I could do about it.
One thing everyone could and should be doing is backups.
Re: SSD will fail at 40k power-on hours (2021)
#28Check your power-on hours:
$ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743
Re: SSD will fail at 40k power-on hours (2021)
#29No post body was provided.
Re: SSD will fail at 40k power-on hours (2021)
#30It's been over two years since this was first identified... since this apparently affected many makes and models of SSDs, it would be nice to know if my laptop could be affected and if there's anything I could do about it.
This will not affect your laptop, all of the models affected by this are enterprise SAS SSDs.
Of course your SSD might have some other firmware bug that would eat your data, all you can do is search for the model number and see if the manufacturer has issued any notices/firmware updates.