Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

221–230 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#221

Post by climb_stealth »

chrischen wrote: post #61
m0llusk wrote: post #35

Earlier quoted context omitted.

Two things: Test restores or you don't actually have backups. Just saying.

I got bit by this with iPhone backups. I did a phone trade in and followed the backup before trading in instructions. Problem is after the trade in the backup failed to restore due to an unknown error. The whole manual syncing and backing up with a cable workflow with Apple is super fickle and riddled with bugs. Luckily I had Time Machine backups of my iOS backups and I managed to avoid losing too much data. As a sid…

Man, Time Machine can fail just as badly. Unknown errors and there is no help or documentation or way to fix it. Carbon Copy Cloner [0] is the way to go for retaining sanity. Absolutely excellent documentation for pretty much any use case. And it works reliably. Not affiliated but after having had terrible experiences with Time Machine I feel compelled to bring it up every time I come across the topic.

[0] https://bombich.com/

Re: SSD will fail at 40k power-on hours (2021)

#222

Post by Denvercoder9 »

abrax3141 wrote: post #214
Denvercoder9 wrote: post #193

Earlier quoted context omitted.

Unlikely; a clock running at 30 Hz overflows a 32 bit (unsigned) integer in 39,768 hours, while the HN disk failed after at least 39,984 hours [1], and the vendors wouldn't issues warnings about 40,000 hours if it actually fails about 230 hours before that. [1] https://news.ycombinator.com/item?id=32031428

Not sure about the mfgr- could be rounding for memory saliency. . . but re HN they’re counting clock time not power on time. Could have easily been turned off total 10 days for maintenance etc. (and good routine could have both servers off about the same no hrs.) That’s only 0.6% downtime.

> Not sure about the mfgr- could be rounding for memory saliency. . .

In that case, I would expect them to be rounding down, not up, and to at least mention the exact number somewhere.

> but re HN they’re counting clock time not power on time. Could have easily been turned off total 10 days for maintenance etc.

I seriously doubt HN has been down for 10 days over the last five years. As discussed in the other thread I linked, this has been one of the longest HN outages.

Re: SSD will fail at 40k power-on hours (2021)

#223

Post by bradfa »

stormbrew wrote: post #144
ratsmack wrote: post #114

Earlier quoted context omitted.

The limiting factor would be the memory chips themselves and any firmware required for them (if any). I also don't know how well they are spec'd and if full documentation is available without NDA's and lawyers.

Looking on mouser and digikey it doesn't seem like flash chips, even into very fairly high density on a single chip[eg. 1], are all that difficult to get and get info on, though they all have very high minimum volume orders. So if a person wanted to try to do this on their own they'd probably be best off finding like 50 friends to go in on the order with them. [1] https://www.mouser.ca/datasheet/2/671/micron_technolo…

The new, fast, high density flash chips from the big name flash chip vendors are not generally even listed on the vendor websites. You have to talk to a sales person and convince them you're actually going to buy in volume to even get data on the latest generation of flash ICs. You will also likely need more than 50 friends to meet the order minimums, unless your 50 friends each want to buy about an ExaByte worth of flash chips.

Also, with these multiple level flash technologies (quad level is current tech, triple is still used in some SSD/NVMe) the read, write, and ECC algorithms are non-trivial to the point where last I checked even mainline Linux's raw flash driver support won't do anything beyond single level cell flashes (and very few new embedded designs are choosing raw parallel NAND flash, instead opting for things like eMMC or UFS which have built-in controllers to handle this).

Re: SSD will fail at 40k power-on hours (2021)

#224

Post by neycoda »

"the SSD will report that 0 GB of available storage space remains. The drive will go offline and become unusable."

Considering this is a firmware "bug" that bricks the drive, due to a misplaced index, not a physical wear issue, it appears to be a 4.5-year planned obsolescence feature.

Re: SSD will fail at 40k power-on hours (2021)

#225

Post by duckmysick »

zh3 wrote: post #187
lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

I seem to have the world's oldest SSD (or am I misinterpreting the output?) (shell 1) ~# smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours_and_Msec 0x0032 000 000 000 Old_age Always - 933932h+27m+33.940s

Fellow time traveller?

Re: SSD will fail at 40k power-on hours (2021)

#226

Post by pmlnr »

kazen44 wrote: post #9
solardev wrote: post #7

Earlier quoted context omitted.

Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…

the chance of two SSD's failing at the same time under normal circumstances is extremely slim. So this might actually be a good cause of this incident.

Heh, no. We had a fleet of HPE Cloudline (CL3100) failing at the same time because the SSDs exhausted the writes.

Re: SSD will fail at 40k power-on hours (2021)

#227

Post by mcv »

mst wrote: post #191
mcv wrote: post #180

Earlier quoted context omitted.

If it's really 4 drives, bought at the same time, failing simultaneously, that's pretty damning evidence. Remind me of my laptop that I bought with 2 SSDs. Not from HP or Dell, but still, I now wonder if I should replace one of them with a more recent SSD and give the other to my son (he currently has an anemic SSD that's too small to install Genshin Impact on).

Assuming you have the budget for it a worst case scenario of "I made my son happy while achieving nothing -else-" doesn't strike me as terrible at all.

It costs money and there's no guarantee it will actually make him happy. It could lead to him playing the game in some dark corner where no one can find him. There are advantages to him having to use the desktop PC.

Re: SSD will fail at 40k power-on hours (2021)

#228

Post by abrax3141 »

Denvercoder9 wrote: post #222
abrax3141 wrote: post #214

Earlier quoted context omitted.

Not sure about the mfgr- could be rounding for memory saliency. . . but re HN they’re counting clock time not power on time. Could have easily been turned off total 10 days for maintenance etc. (and good routine could have both servers off about the same no hrs.) That’s only 0.6% downtime.

> Not sure about the mfgr- could be rounding for memory saliency. . . In that case, I would expect them to be rounding down, not up, and to at least mention the exact number somewhere. > but re HN they’re counting clock time not power on time. Could have easily been turned off total 10 days for maintenance etc. I seriously doubt HN has been down for 10 days over the last five years. As discussed in the other thread I…

Yeah, no, I don’t mean crashed altogether, just off for routine maintenance on a rotating basis. But anyway, no sense in prolonging this. Anyway, agree that 1% seems high.

Re: SSD will fail at 40k power-on hours (2021)

#229

Post by ekidd »

taneq wrote: post #134
mkl wrote: post #11

Earlier quoted context omitted.

Not two SSDs, four : two in the main server, and two in the backup server.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

I have definitely seen RAID arrays where the drives were all part of a single manufacturing batch, and multiple drives all failed in rapid succession. I think this can be caused by several things:

- Unless you periodically do full-drive reads, you may silently accumulate bad blocks across multiple drives in an array. When you finally detect a failed drive, you discover that other drives have also been failing for months.

- A full RAID rebuild is a high-stress event that tries to read every disk block on every drive, as rapidly as possible.

- And finally, some drive batches are just dodgy, and it may not take much to push them over. And if identically dodgy drives are all exposed to exactly the same thermal stress and the same I/O operations, then I guess they might fail close together?

Honestly, RAID arrays only buy you so much reliability. Hardware RAID controllers are another single point of failure. I once lost two drives and a RAID controller all together during Christmas, which was not a fun time.

I do like the modern idea of S3-like storage, where data is replicated over several independent machines, and the controlling software can recover from losing entire servers (or even data centers). It's not a perfect match for everything, but it works great for lots of things.

Re: SSD will fail at 40k power-on hours (2021)

#230

Post by bluedino »

taneq wrote: post #134
mkl wrote: post #11

Earlier quoted context omitted.

Not two SSDs, four : two in the main server, and two in the backup server.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

Or, update the firmware on your switches, servers, drives, etc on a regular basis.
Post reply on HN