Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSD will fail at 40k power-on hours (2021)

cisco.com

Post reply on HN
Print view View original

191–200 of 276 posts

Re: SSD will fail at 40k power-on hours (2021)

#191

Post by mst »

mcv wrote: post #180
dredmorbius wrote: post #87

Earlier quoted context omitted.

It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.

If it's really 4 drives, bought at the same time, failing simultaneously, that's pretty damning evidence. Remind me of my laptop that I bought with 2 SSDs. Not from HP or Dell, but still, I now wonder if I should replace one of them with a more recent SSD and give the other to my son (he currently has an anemic SSD that's too small to install Genshin Impact on).

Assuming you have the budget for it a worst case scenario of "I made my son happy while achieving nothing -else-" doesn't strike me as terrible at all.

Re: SSD will fail at 40k power-on hours (2021)

#192

Post by bo0tzz »

zh3 wrote: post #187
lucb1e wrote: post #28

Check your power-on hours: $ sudo smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9743 Just looking at the raw value, it seems to be 9'743 hours in my case

I seem to have the world's oldest SSD (or am I misinterpreting the output?) (shell 1) ~# smartctl -a /dev/sda | grep -e Power_On_Hours -e ^ID ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 9 Power_On_Hours_and_Msec 0x0032 000 000 000 Old_age Always - 933932h+27m+33.940s

That would make for 106 years of power on time, so it's probably not right...

Re: SSD will fail at 40k power-on hours (2021)

#193

Post by Denvercoder9 »

abrax3141 wrote: post #173

4 byte integer rollover: 2^32=4,294,967,296/4.5=954,437,177/365=2,614,896/3600=726/24=30ticks/sec

Unlikely; a clock running at 30 Hz overflows a 32 bit (unsigned) integer in 39,768 hours, while the HN disk failed after at least 39,984 hours [1], and the vendors wouldn't issues warnings about 40,000 hours if it actually fails about 230 hours before that.

[1] https://news.ycombinator.com/item?id=32031428

Re: SSD will fail at 40k power-on hours (2021)

#194

Post by kqr »

dredmorbius wrote: post #167
kqr wrote: post #152

Earlier quoted context omitted.

> I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or ev…

My point is that risk is central to systems management. If you look at earlier standard texts on the subject, e.g., Nemeth or Frisch, the concept of risk is all but entirely missing. I've numberous disagreements with Google, but one place where I agree is that the term SRE, systems reliability engineer, puts the notion of managing for stability front and centre, and inherently acknowledges the principle of risk. I've…

Okay, fair enough. To me, the risk involvement is so obvious (to any serious business function!) that I find the management/reduction distinction a more important point. But I can see it your way too.

Re: SSD will fail at 40k power-on hours (2021)

#195

Post by helionsantos »

charcircuit wrote: post #13
fartcannon wrote: post #10

Earlier quoted context omitted.

Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you s…

Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?

Considering immoral practices adopted by corporations, such as vendor lock-in, use of slave work (directly or indirectly), law bending for its own interests, supporting and conducting biased research towards its own interests among others, I would say that is quite sensible to believe it. Big corporations, per se, are not evil entities. The people running them might or might not be, and when you have evil/immoral people running things, unless there are good control measures in place, they might take bad decisions.

Re: SSD will fail at 40k power-on hours (2021)

#196

Post by bobsmooth »

mywacaday wrote: post #189
taneq wrote: post #134

Earlier quoted context omitted.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

About 20 years ago I worked for a small storage company, the person that managed the returns of disks form customers was very strongly of the opinion that the odd firmware versions on Seagate drives were returned way more often than the even.

That's the kind of superstition that's only brought about by deep trauma.

Re: SSD will fail at 40k power-on hours (2021)

#197

Post by mst »

dredmorbius wrote: post #167
kqr wrote: post #152

Earlier quoted context omitted.

> I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or ev…

My point is that risk is central to systems management. If you look at earlier standard texts on the subject, e.g., Nemeth or Frisch, the concept of risk is all but entirely missing. I've numberous disagreements with Google, but one place where I agree is that the term SRE, systems reliability engineer, puts the notion of managing for stability front and centre, and inherently acknowledges the principle of risk. I've…

A friend of mine who was once at Google mentioned that in a meeting once their SRE group was told by their senior manager roughly "you've broken stuff in $time_period far less than your outage budget, and that probably means you should've been rolling out features faster".

Risk -levels- are a choice and also a trade-off.

Re: SSD will fail at 40k power-on hours (2021)

#198

Post by TeamMCS »

taneq wrote: post #134
mkl wrote: post #11

Earlier quoted context omitted.

Not two SSDs, four : two in the main server, and two in the backup server.

Yowch. The old "stagger your drive replacements, stagger your batches" thing might not be quite as outdated as we'd like to think...

Plus you mitigate getting a bad batch of the same drive. See IBM Deathstars and the Seagate drives.

May have to go check my up hours on my drives now, I must have a few nearing that sort of write hours

Re: SSD will fail at 40k power-on hours (2021)

#200

Post by dredmorbius »

kqr wrote: post #194
dredmorbius wrote: post #167

Earlier quoted context omitted.

My point is that risk is central to systems management. If you look at earlier standard texts on the subject, e.g., Nemeth or Frisch, the concept of risk is all but entirely missing. I've numberous disagreements with Google, but one place where I agree is that the term SRE, systems reliability engineer, puts the notion of managing for stability front and centre, and inherently acknowledges the principle of risk. I've…

Okay, fair enough. To me, the risk involvement is so obvious (to any serious business function!) that I find the management/reduction distinction a more important point. But I can see it your way too.

Thanks.

I don't know how long you've been in the business, but the change seems a relatively recent one, one that wasn't manifestly obvious to me, and one that has pretty much always seemed difficult to communicate to management.

Whether that's because business management is often about ignoring risks or treating it as inconvenient, or if I've just had a long string of bad bosses, I'm not sure.

I did make a point of looking through several of the books that were formative for me (mostly 1990s and 2000s publication dates), and there's little addressing the point. Limonchelli's book on time management for sysadmins was a notable departure from the standard when it came out, in 2008. I'd say that marked the shift toward structured and process-oriented practices.

That was about the time of the transition from "pets" to "cattle" (focus on individual servers vs. groups / farms), but pre-dates the cloud transition.

Post reply on HN