Why I'm usually unnerved when modern SSDs die on us • Hacker News Board

Re: Why I'm usually unnerved when modern SSDs die on us

#221

Post by shittyadmin » Wed, Dec 12, 2018, 3:44 AM UTC

SSLy wrote: post #178

shittyadmin wrote: post #2

I've experienced a few seriously strange issues with modern SSDs, even some of the better ones. I had a 512GB Samsung drive that became very slow randomly at doing IO operations, the whole machine would die for 10-30 seconds at a time once or twice a day while any process that tried to use the disk became blocked on IO. Then it'd come right back like everything was perfectly fine. Issues like this definitely worry me…

Linux? Try halving down your page caches.

This was on my Windows gaming machine.

Re: Why I'm usually unnerved when modern SSDs die on us

#222

Post by nickpsecurity » Wed, Dec 12, 2018, 3:59 AM UTC

bacon_waffle wrote: post #192

monocasa wrote: post #184

Earlier quoted context omitted.

For those doubting you, the going rate for embedded engineers out here in the Denver area where a lot of these SSD controllers are designed is ~$90k. Embedded engineers get peanuts for some reason.

I've noticed the same thing, and suspect that it's related to the way that higher level software scales compared to embedded. If a line of code is written to run in a customer's browser, then that line of code may be deployed to millions, maybe billions, of customers. But, if an equivalent line of code goes in to firmware for some widget, then you're doing pretty good to get that line in to a million widgets at all,…

Nah, it's just management incentives with nothing on liability side for the product countering that. They keep labor costs low. The company still makes piles of money off the product. There's no liability for devices, esp cheap, that fail randomly after such and such period of time. So, no negative outcomes for keeping labor cost down for firmware. So, they keep doing it.

That simple.

Re: Why I'm usually unnerved when modern SSDs die on us

#223

Post by nickpsecurity » Wed, Dec 12, 2018, 4:03 AM UTC

xyzzy_plugh wrote: post #101

loeg wrote: post #93

Earlier quoted context omitted.

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not…

Both, and usually significantly. Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazi…

"Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. "

Heaven for generalists that always love doing new kinds of things. Once I learned about it, I knew I probably should've done embedded instead of security research. Of course, now there's significant interest in overlap. Might not be too late to learn all that stuff after all. :)

Re: Why I'm usually unnerved when modern SSDs die on us

#224

Post by feelix » Wed, Dec 12, 2018, 4:16 AM UTC

pkaye wrote: post #91

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all th…

Sorry to spin off topic, but perhaps having the chance to speak to someone that's worked on SSD firmware for the first time...

is there any feasible way to recover data after a TRIM command has been issued that you can think of? Is there any way to trick the firmware into not returning 0's when reading the blocks of a deleted file? Mostly interested in doing so for Apple

TRIM destroying the entire data recovery and forensics market seems like such a big deal, I still can't believe it although it started years ago

Re: Why I'm usually unnerved when modern SSDs die on us

#225

Post by bacon_waffle » Wed, Dec 12, 2018, 4:18 AM UTC

londons_explore wrote: post #207

bacon_waffle wrote: post #192

Earlier quoted context omitted.

I've noticed the same thing, and suspect that it's related to the way that higher level software scales compared to embedded. If a line of code is written to run in a customer's browser, then that line of code may be deployed to millions, maybe billions, of customers. But, if an equivalent line of code goes in to firmware for some widget, then you're doing pretty good to get that line in to a million widgets at all,…

Write an embedded bootloader and there's a good chance your code will be used by a billion people within a few years of writing it. You'll still get paid peanuts for doing it.

Yes, and that's just it.

If a software product takes off, that can happen incredibly quickly, and the new product is primarily composed of code. If a hardware product takes off, the change can't be nearly as fast as it's bound by manufacturing, and the code is just one component in each thing.

I happen to be working on firmware for a VOIP phone today - we'll end up making N million of these things, over some number of years. If I were working on an Android app with similar functionality, that app could conceivably go to N million people tomorrow, or 10N, or 100N...

Anyway, I don't think I have a particularly clear or concise (or even correct) argument here, but it's the only way I've been able to rationalise what we've observed.

Re: Why I'm usually unnerved when modern SSDs die on us

#226

Post by romland » Wed, Dec 12, 2018, 4:37 AM UTC

cm2187 wrote: post #36

HelloNurse wrote: post #17

TL;DR Lack of noises makes SSD drives bad at motivating users to do backups or use redundant storage: they don't seem to be on the verge of catastrophic failure.

I think operating systems should be programmed to wipe out a drive completely once early in the life of every user (around age 20-ish) to burn in their brain the need to back up!

My mom always told me that once I've fallen off horses one hundred times, I will never fall off again.

(I filed a request for a bike in my early life, so never got to that 100).

Re: Why I'm usually unnerved when modern SSDs die on us

#227

Post by Serow225 » Wed, Dec 12, 2018, 4:40 AM UTC

blattimwind wrote: post #115

hwillis wrote: post #100

Earlier quoted context omitted.

I do not believe that. 1. I can't find a source that says less than a few nanometers. 2. 300 picometers is roughly the diameter of a helium diatom. The head cannot possibly float through hydrodynamic means if an air molecule can barely even fit under it.

> The head cannot possibly float through hydrodynamic means if an air molecule can barely even fit under it. It can. Since siblings liked airplane analogies, here is another one: Consider the head to be an airplane. It has somewhat wing-similar features which provide a lifting force, but the actual read/write head sits below those features (like, say, a landing gear is below wings).

This is correct :)

Re: Why I'm usually unnerved when modern SSDs die on us

#228

Post by nickpsecurity » Wed, Dec 12, 2018, 4:43 AM UTC

rsync wrote: post #41

Waterluvian wrote: post #5

"When a HD died early, you could also imagine undetected manufacturing flaws that finally gave way. With SSDs, at least in theory that shouldn't happen" Why shouldn't it? Isn't it just hardware too? "With spinning HDs, drives might die abruptly but you could at least construct narratives about what could have happened to do that" Why can't you do the same with SSDs? It feels like the author's main complaint is the fr…

"It feels like the author's main complaint is the frustration of not understanding SSD hardware as well." What is so frustrating about SSDs is how very poorly they compare to previous incarnations of solid state storage. Using Disk-On-Chip and/or IDE-pin-compatible CF cards, I had many, many devices in the field that lasted, mounted read-only, for decades An entire sect of the computing industry came to rely on these…

"We shouldn't have to do that."

I don't know about that. I know most managers in embedded software go cheap on engineers and software assurance on purpose to get bonuses and such. The hardware side makes me think you haven't studied deep, sub-micron hardware much. I started looking into it a few years ago or so, just reading the slides on lots of stuff even though not understanding much of it. They helpfully put a lot in lay terms, though, with lots of comparisons. To say the stuff gets harder every time you shrink to a smaller node is an understatement, esp for solid state.

If anything, the modern flash should probably be considered broken right as they ship out of the factory. If not, the process nodes after 90nm or so just keep having more and more ways for individual components to screw up or change behavior across same wafer. Some happen instantly by design. Some happen later with aging. The memory technologies are closer to that analog level of things than most with harder verification. The high-density flash on newer nodes uses less-reliable tech than most just to operate at that low cost. So, they add all kinds of firmware and hardware tricks to try to make it work for a period of time like it's a whole, functional unit of storage despite pieces of it misbehaving all throughout.

It's a nice, man-made miracle these techs even work at all. Those that last longer like you mentioned still exist. I'll add a comment with links to one type so you can compare price/storage/performance to these broken-by-design SSD's you use. I'll throw in another two that mention shrinking challenges so you can see what they face every time they have to upgrade or just deploy new designs in mixed-signal.

https://news.ycombinator.com/item?id=18528023

https://www.electronicdesign.com/digital-ics/understanding-2...

https://anysilicon.com/ic-design-impact-in-moving-from-28nm-...

Re: Why I'm usually unnerved when modern SSDs die on us

#229

Post by linsomniac » Wed, Dec 12, 2018, 4:43 AM UTC

From reading that blog and it's sister post about "flaky SMART data" on those same Crucial MX500 drives, reminds me that not all SSDs are created equal.

Just like not all hard drives are created equal. My previous job involved a decade running 10 cabinets of servers an hour away with very little manpower: we eventually came to find that IBM/HGST drives were a lot more reliable than others.

We also evaluated some early SSDs, and they were terribly unreliable. We eventually settled on the Intel drives and they were superb. My new job we've been using mostly Intel and Samsung Pro drives, they work great. But Dell sent us a server with some "enterprise SSDs" in it, that we eventually found were Plextor drives. Those things were terrible. We replaced them immediately with Intel, but used some of the Plextor drives and had all of them fail within a year. I'd put the Intel 64GB SLC drives from our 7 year old database server in a system before I'd put one of those brand new "enterprise" Plextor drives in.

I love Crucial, I buy a lot of RAM from them, but I'm skeptical of switching to other brands of SSDs. The more experience I have, the more conservative I get with systems that matter.

Re: Why I'm usually unnerved when modern SSDs die on us

#230

Post by Serow225 » Wed, Dec 12, 2018, 4:46 AM UTC

loeg wrote: post #161

blattimwind wrote: post #117

Earlier quoted context omitted.

It doesn't need to be. The basic idea is that the fly height is self-regulating; if the head goes away from the platter, its "lift" is reduced, so the springiness of the arm forces it back to the platter. If it moves closer to the platter, lift is increased, so it moves away. Similarly the tracks don't have to be perfectly round or concentric, because the lowest-level head control system in the drive actively tracks…

> Similarly the tracks don't have to be perfectly round or concentric They have to be pretty dang close at 7200+ rpm.

Nope. Since a long time, tracks are virtual (array of raw track IDs as a function of sector ID when the arm is held "constant" and averaged for X thousand RPMs as part of the per-drive firmware Calibration at the factory).