Live data from Hacker News

Why I'm usually unnerved when modern SSDs die on us

utcc.utoronto.ca

101–110 of 258 posts

Re: Why I'm usually unnerved when modern SSDs die on us

#101
post #93
post #91

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all th…

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not…

Both, and usually significantly.

Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer.

Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazing.

I left the embedded space and have never looked back.

Re: Why I'm usually unnerved when modern SSDs die on us

#102
post #43

Earlier quoted context omitted.

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

I do not believe that. 1. I can't find a source that says less than a few nanometers. 2. 300 picometers is roughly the diameter of a helium diatom. The head cannot possibly float through hydrodynamic means if an air molecule can barely even fit under it.

Neat, you independently discovered why the industry is moving to helium-filled drives.

Re: Why I'm usually unnerved when modern SSDs die on us

#103

Earlier quoted context omitted.

To be fair, the 'I' in RAID stands for 'inexpensive'.

I think a lot of people know the I as "Independent" these days since RAID is usually an enterprise feature with enterprise costs.

Modern takes on RAID are built in to home OSs these days, and consumer grade NAS devices are pretty common. It really isn't a fancy enterprise-only technology at all.

Re: Why I'm usually unnerved when modern SSDs die on us

#104
post #93

Earlier quoted context omitted.

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not…

As someone that identified a bug in a Drobo firmware once and was offered a job on the spot I think the problem with attracting talent is two fold. The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion. Kids don't routinely grow up with a SAN in the house. They do tend to…

I'm mostly curious because I work in a storage-adjacent field (NAS) for a BigCorp and the pay is pretty good, if not quite FAANG level. It's not a startup by any means, but I will easily become a multi-millionaire in a handful of years. I was curious about the other side of the fence.

Re: Why I'm usually unnerved when modern SSDs die on us

#106
post #93

Earlier quoted context omitted.

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not…

Both, and usually significantly. Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazi…

> Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer.

Yeah, yikes. That does not sound appealing :-).

Re: Why I'm usually unnerved when modern SSDs die on us

#107
post #43

Earlier quoted context omitted.

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

I do not believe that. 1. I can't find a source that says less than a few nanometers. 2. 300 picometers is roughly the diameter of a helium diatom. The head cannot possibly float through hydrodynamic means if an air molecule can barely even fit under it.

Bryan Cantrill made this claim in a talk recently[1], and the fly-height is apparently 800 picometers.

[1]: https://youtu.be/fE2KDzZaxvE?t=1551

Re: Why I'm usually unnerved when modern SSDs die on us

#108
post #63

This is not a technological problem, it's a cultural one. These problems are easily fixed ("easily" by the standards of technical problems that regularly get fixed in other regimes). The reason they don't get fixed is that the customer reaction to failures like this is to rant at the mysterious storage gods that are making their lives miserable. Needless to say, there are no mysterious storage gods. These are artifac…

Physical devices don't have SLAs, because they aren't services. SSDs have the physical equivalent of an SLA, a warranty. I haven't seen stats, but in my experience the odds of an SSD dying within the warranty period is very very low. If you want your storage to have an SLA, storage service providers exist, and will be happy to give you an SLA if you're willing to pay. But it isn't cheap.

> Physical devices don't have SLAs

Yes, I know that. That's why I said "essentially".

> SSDs have the physical equivalent of an SLA, a warranty.

There are two orthogonal issues. The first is what happens when a device fails. A warranty addresses that. The second is how does it fail. Does it fail all at once with no warning, no way to perform post-mortem diagnostics, and no way to recover the data? Or does it fail with a gradual degradation of performance and capacity over time, and in a way that, if/when total failure occurs, the cause can be ascertained and the data still recovered somehow?

Re: Why I'm usually unnerved when modern SSDs die on us

#109
post #91

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all th…

This is a bit of a misrepresentation. The only flash that never had ECC was NOR. Some embedded systems had NOR, but I'd be incredibly rare to find a consumer SSD with NOR.

When designing a NAND memory product, you aim for some max allowed error rate. You choose the error correction algorithm based on that target. Error rates for NAND products are precisely what the designer intended.

Because SSDs are so large and there is such a large number of them, errors can still occur (at a known rate). FTLs should take that into account. Critical data structures can add checksum redundancy which can reduce the error rate for those to an even lower value, which is usually necessary anyway since power disturbances during erase or program can cause programming errors.

There are of course a number of patterns that increase error rates that FTLs have to be programmed to prevent. Encrypting is the first important step, since it makes the data unlikely to be uniform. The second is mitigating read disturb.

Re: Why I'm usually unnerved when modern SSDs die on us

#110
post #21

Not that spinning HDDs are really any different, but SSDs are a perfect example of an entire computer that you attach to yours, and speak with through one of the (many) storage-oriented protocols. The device itself is a black box, and complex transformations take place between the physical persistence of the data and the logical structures that are exchanged on the wire. There are many layers of indirection, and many…

> These mechanical components exhibit wear proportional to their use

Actually the spindle isn't touching anything any more, because the spindle/rotor (one part) is supported by a fluid bearing; it basically floats on a thin layer of oil. If the spindle touches the bearing at, essentially any speed that isn't zero, the bearing surfaces are damaged instantly and the resulting burrs and debris will degrade and lock up the bearing very quickly.

I believe the only rolling-element/contact bearing used in modern disks is the pivot bearing of the arm assembly.

Post reply on HN