Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I'm usually unnerved when modern SSDs die on us

utcc.utoronto.ca

Post reply on HN
Print view View original

91–100 of 258 posts

Re: Why I'm usually unnerved when modern SSDs die on us

#91

Post by pkaye »

I worked on SSD firmware for quite a long time and here is my perspective.

Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee...

To compensate for all these deficiencies, the SSD architecture and hence the entire FTL becomes very complicated because any part of it can become damaged at any time. We always have to have backup algorithms to recovery from any scenario. Its difficult to build algorithms that can recovery from arbitrary failures in a reasonable time. I cannot have a drive sitting around for 20 minutes trying to fsck itself.

Another problem is that the job while rewarding is not very lucrative. The chance of a multi million dollar payoff for an employee is low. I have a higher chance working on a web connected gadget to become a millionaire. So that means it is really hard to recruit those who are top notch programmers who known how to figure out the algorithms, write the code, debug the hardware. Most new grads these days are interested in python, javascript and machine learning.

Re: Why I'm usually unnerved when modern SSDs die on us

#92

Post by Misdicorl »

cyphar wrote: post #43
niftich wrote: post #21

Not that spinning HDDs are really any different, but SSDs are a perfect example of an entire computer that you attach to yours, and speak with through one of the (many) storage-oriented protocols. The device itself is a black box, and complex transformations take place between the physical persistence of the data and the logical structures that are exchanged on the wire. There are many layers of indirection, and many…

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

This seems implausible at first glance. Even making objects flat at the nanometer scale is quite difficult. When you also introduce movement tolerances (e.g. rotation disk is not perfectly aligned), this seems quite extraordinary.

Can you provide some references?

Re: Why I'm usually unnerved when modern SSDs die on us

#93

Post by loeg »

pkaye wrote: post #91

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all th…

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens.

> Another problem is that the job while rewarding is not very lucrative.

Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?

Re: Why I'm usually unnerved when modern SSDs die on us

#94

Post by cyphar »

Misdicorl wrote: post #92
cyphar wrote: post #43

Earlier quoted context omitted.

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

This seems implausible at first glance. Even making objects flat at the nanometer scale is quite difficult. When you also introduce movement tolerances (e.g. rotation disk is not perfectly aligned), this seems quite extraordinary. Can you provide some references?

(Sheepishly) My only reference is a talk by Bryan Cantrill[1] where he states that an exec at a hard-drive manufacturer stated it was ".8 nanometers". I will try to find a better source than that.

EDIT: There is a paper from 2016 which did an analysis in the difference in the flying height of the head during different operations, and it was measured in Angstroms[2]. I couldn't find one that actually gives a precise value of the flying height. There is a 2011 paper which states that some system they were testing allowed for 4-9 nm flying heights[3] which is about half-an-order-of-magnitude larger than the claim -- but that's already 7 years old.

[1]: https://youtu.be/fE2KDzZaxvE?t=1551 [2]: http://iieng.org/images/proceedings_pdf/E0116009.pdf [3]: http://maeresearch.ucsd.edu/callafon/publications/2011/UweIE...

Re: Why I'm usually unnerved when modern SSDs die on us

#95

Post by loeg »

niftich wrote: post #21

Not that spinning HDDs are really any different, but SSDs are a perfect example of an entire computer that you attach to yours, and speak with through one of the (many) storage-oriented protocols. The device itself is a black box, and complex transformations take place between the physical persistence of the data and the logical structures that are exchanged on the wire. There are many layers of indirection, and many…

I don't really agree with most of this comment. HDDs are also inscrutable black boxes; many of their failures are controller, rather than media losses; and SSDs also report SMART attributes that are predictive of failure. It's certainly possible HDD vendors have done a more successful job of convincing buyers that failures are attributable to the media rather than the controller, but utilizing the media fully with shingled recording and HAMR and all that jazz really requires a similar degree of controller complexity as an SSD FTL controller.

Re: Why I'm usually unnerved when modern SSDs die on us

#96

Post by loeg »

Misdicorl wrote: post #92
cyphar wrote: post #43

Earlier quoted context omitted.

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

This seems implausible at first glance. Even making objects flat at the nanometer scale is quite difficult. When you also introduce movement tolerances (e.g. rotation disk is not perfectly aligned), this seems quite extraordinary. Can you provide some references?

Disk platters rotating at 7200 rpm are both perfectly smooth and perfectly aligned.

The read head is something like a jumbo jet flying a handful of feet above the (perfectly smooth) ground. It's really crazy how close these things are, moving very fast. And why accelerometers are a significant feature.[0]

> Can you provide some references?

Wikipedia claims[1]:

> In 2011, the flying height in modern drives was a few nanometers.

and

> The "flying height" is constantly decreasing to enable higher areal density.

> At 7,200 RPM, the edge of the platter is traveling at over 120 kilometres per hour (75 mph)

So you've got a read head flying at 120 km/h => 33 m/s => 33,000,000,000 nm/s at a height of 3nm or less. Picture that!

E.g, a 757 typically cruises at 858 km/h => 238 m/s. So picture your 757 flying at an altitude of 21 nm and that's the metaphor, kinda. The read head is a bit smaller than 1/7 of a 757 jet, obviously.

[0]: https://en.wikipedia.org/wiki/Head_crash

[1]: https://en.wikipedia.org/wiki/Flying_height

Re: Why I'm usually unnerved when modern SSDs die on us

#97

Post by notatoad »

lisper wrote: post #63

This is not a technological problem, it's a cultural one. These problems are easily fixed ("easily" by the standards of technical problems that regularly get fixed in other regimes). The reason they don't get fixed is that the customer reaction to failures like this is to rant at the mysterious storage gods that are making their lives miserable. Needless to say, there are no mysterious storage gods. These are artifac…

Physical devices don't have SLAs, because they aren't services. SSDs have the physical equivalent of an SLA, a warranty. I haven't seen stats, but in my experience the odds of an SSD dying within the warranty period is very very low.

If you want your storage to have an SLA, storage service providers exist, and will be happy to give you an SLA if you're willing to pay. But it isn't cheap.

Re: Why I'm usually unnerved when modern SSDs die on us

#98

Post by starbeast »

pinebox wrote: post #42

I actually much prefer this SSD failure mode: Unlike failing spinning rust which will happily linger around coughing up bad data (which will then be written to backups, mirrored drives, etc. potentially creating a huge mess) an SSD going out like a light is comfortingly binary.

Is the thorny problem of elegant vs graceful degradation. In a raid system you want something elegant but not synchronized. In a single drive, some sort of graceful degradation is usually preferable.

http://www.assetinsights.net/Glossary/G_Elegant_Degradation....

http://www.assetinsights.net/Glossary/G_Graceful_Degradation...

Re: Why I'm usually unnerved when modern SSDs die on us

#99

Post by godzillabrennus »

loeg wrote: post #93
pkaye wrote: post #91

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all th…

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not…

As someone that identified a bug in a Drobo firmware once and was offered a job on the spot I think the problem with attracting talent is two fold.

The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion.

Kids don't routinely grow up with a SAN in the house. They do tend to grow up with lots of internet connected consumer caliber devices and can easily gain exposure to working with these technologies.

I was fortunately able to explore this type of technology in depth because a family owned business let me tinker with their server equipment in high school.

After college I then co-founded a startup back before the cloud became big. That meant we needed to make use of old hardware to provide service to our customers at a price point that made our service profitable. Old drives were not a reliable way to do that. New drives were extremely expensive for old servers back in the day when SCSI was the interface that you expected for a server. We had to get creative and play with JBOD devices. ZFS was an amazing tool for us in those days, and it still is for anyone who wants to tinker.

The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff.

Knowledge is power though. I recently came across an AI startup that I'm now helping. They were spending significant money using GPU computational power to provide artificial intelligence training through a cloud provider. They blew through about $300k in credits within the first year to give you an idea of how much money that type of power can cost.

I am now helping them cut over to their own co-location facility. The first year alone they will save so much money it will pay for the next three years.

Then you read articles like this: https://www.newyorker.com/magazine/2018/12/10/the-friendship...

Reading that helps reinforce the idea that no matter what path you are on in the field there is potential that some random thing you learned about an SSD firmware helps you optimize some growth stage companies product and ultimately that helps you build wealth.

Re: Why I'm usually unnerved when modern SSDs die on us

#100

Post by hwillis »

cyphar wrote: post #43
niftich wrote: post #21

Not that spinning HDDs are really any different, but SSDs are a perfect example of an entire computer that you attach to yours, and speak with through one of the (many) storage-oriented protocols. The device itself is a black box, and complex transformations take place between the physical persistence of the data and the logical structures that are exchanged on the wire. There are many layers of indirection, and many…

> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 pic…

I do not believe that.

1. I can't find a source that says less than a few nanometers.

2. 300 picometers is roughly the diameter of a helium diatom. The head cannot possibly float through hydrodynamic means if an air molecule can barely even fit under it.

Post reply on HN