Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

11–20 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#11

Post by mkl »

solardev wrote: post #7
dredmorbius wrote: post #2

Possibly related to recent HN issues, see: https://news.ycombinator.com/item?id=32031243

Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…

Not two SSDs, four: two in the main server, and two in the backup server.

Re: SSD will fail at 40k power-on hours (2021)

#12

Post by dr_zoidberg »

usr1106 wrote: post #3

Cisco is not a SSD manufacturer. They write industry-wide bug. Does that mean that more than one SSD manufacturer is affected (because they use partially the same firmware)? Further down they mention only Sandisk. Or is the industry-wide just their newspeak for saying any Sandisk of affected model, regardless whether installed in a Cisco box or somewhere else?

I'm interested here too. I've got a Crucial SSD from 2015 that's been on about:

* 100% of 2015-2017, let's add 2 years here

* Aboutish 50% of days since 2018 to 2020

* On and off again (5%?) since then until now.

So it's about 3 years of full use? I'm eyeballing the use here. So it may be close to the numbers that were given, but I'm not sure. Guess I could check the SMART stats to get a precise number and from there decide what to do about it.

Searching a bit it seems it's a well-known bug in "enterprise SSDs"[0, 1] (which my drive certainly isn't) but there aren't any real details about what causes it, other than "a firmaware bug".

[0] https://www.servethehome.com/hpe-issues-hpd7-fix-for-ssds-th...

[1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...

Re: SSD will fail at 40k power-on hours (2021)

#13

Post by charcircuit »

fartcannon wrote: post #10
twawaaay wrote: post #5

Earlier quoted context omitted.

Very strange understanding of the word "evidence". No sane SSD manufacturer would do such thing on purpose. You do it and you loose business, that's it. The simplest explanation is that somebody made an honest engineering mistake.

Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you s…

Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?

Re: SSD will fail at 40k power-on hours (2021)

#14

Post by mkl »

AaronFriel wrote: post #8
justinsaccount wrote: post #6

Earlier quoted context omitted.

Someone pointed out on the other thread that it could be 2^57 nanoseconds: >>> 2**57/10**9/3600 40031.996687737745

If it were 53, I'd wonder "are they storing the time in the integer part of a double precision float?" That wouldn't go negative, it'd just start absorbing increments without changing the value. Though that might cause a divide by zero? What could cause unexpected behavior at 57 bits? Perhaps storing fractions of an hour, like incrementing it every 1/16th of an hour and calculating a relative rate of change, causing…

Do embedded CPUs like the one in an SSD have floating point units? It seems more likely to me that the upper bits in a 64 bit integer counter were used for something else.

Re: SSD will fail at 40k power-on hours (2021)

#16

Post by [deleted] »

charcircuit wrote: post #13
fartcannon wrote: post #10

Earlier quoted context omitted.

Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you s…

Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?

[deleted]

Re: SSD will fail at 40k power-on hours (2021)

#17

Post by alliao »

charcircuit wrote: post #13
fartcannon wrote: post #10

Earlier quoted context omitted.

Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you s…

Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?

planned obsolescence is quite a thing...?

Re: SSD will fail at 40k power-on hours (2021)

#19

Post by danielheath »

AaronFriel wrote: post #8
justinsaccount wrote: post #6

Earlier quoted context omitted.

Someone pointed out on the other thread that it could be 2^57 nanoseconds: >>> 2**57/10**9/3600 40031.996687737745

If it were 53, I'd wonder "are they storing the time in the integer part of a double precision float?" That wouldn't go negative, it'd just start absorbing increments without changing the value. Though that might cause a divide by zero? What could cause unexpected behavior at 57 bits? Perhaps storing fractions of an hour, like incrementing it every 1/16th of an hour and calculating a relative rate of change, causing…

Packing a type flag into the upper bits of a 64 bit value is a reasonably common optimisation in dynamic language implementations (because it lets you use unboxed number arithmetic).

Re: SSD will fail at 40k power-on hours (2021)

#20

Post by bayindirh »

twawaaay wrote: post #5
userbinator wrote: post #4

40000 (or even 40960) seems an odd number to fail at. 64k or 32k would make the cause pretty obvious, but 40000 doesn't seem all that round in binary. Perhaps a 12-bit counter incrementing every 10h? This is puzzling. Of course, I am also entertaining the possibility that no one thought they would be in use for this long, which would certainly be evidence of planned obsolescence.

Very strange understanding of the word "evidence". No sane SSD manufacturer would do such thing on purpose. You do it and you loose business, that's it. The simplest explanation is that somebody made an honest engineering mistake.

When you purchase a server (fleet), you get a long warranty with it. Generally 3 to 5 years. So you expect this fleet to stay in service for Unless you burn through your SSDs, you're very unlikely to hit this event.

When these servers' continue to be used and disks all start to fail at the same time, this will obviously stink.

The bathtub curve is not like this. You can feel that.

Post reply on HN