Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs: A gift and a curse

laur.ie

Post reply on HN
Print view View original

81–90 of 99 posts

Re: SSDs: A gift and a curse

#81

Post by rosser »

userbinator wrote: post #77
rosser wrote: post #31

Earlier quoted context omitted.

Turns out, the Crucial drives' "power loss protection" doesn't actually preserve data "in-flight" at the moment of power loss. It just prevents data "at rest" from being corrupted. This appears to apply to all the consumer-grade Crucial drives. See [1], money quote: "In the MX100 review, I was still under the impression that there was full power-loss protection in the drive, but my impression was wrong. The client-le…

That is true of ordinary HDDs too. Until data is written to the flash or platter itself, it hasn't been actually written. People don't expect files they haven't saved to be written and magically come back later if the power goes out, but they do expect that their drives will power back up with all the data that was last written intact.

Yes, that's true.

However, Crucial explicitly marketed their SSDs as having, quote, "power loss protection", pointing out the array of capacitors on the drive's PCB, and strongly implying that this feature included in-flight writes surviving a power loss (i.e., that the caps had enough capacity to enable flushing the DRAM buffer to the NAND media, much like the non Sandforce-based Intel SSDs — e.g., the 320 series from a few years back, or the current DC-3500 and -3700 drives).

That, it turns out, isn't true.

And that's a problem, because I and many others bought these drives on the basis of Crucial's implying they were power-loss durable. When enough people started reporting to Crucial support that their drives didn't, in point of actual fact, offer this feature, their marketing literature changed so as not to imply they did, and forum posts and reviews, like my previously-linked Anand Tech article, started pointing out that they didn't.

Re: SSDs: A gift and a curse

#82

Post by semi-extrinsic »

dragontamer wrote: post #76
mrb wrote: post #10

Of course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash chips on a PCB, take the controller's manufacturer's reference firmware implementation, tweak the dozens of knobs provided by the reference implementation (ignore FLUSH commands, change the amount of reserved sectors, disable this, enable that, etc), change th…

50%+ failure rate on 3TB Seagates from Backblaze studies. Something _horrible_ happened on the 3TB Seagate batch. 4TB and 2TB are reasonable... but 3TB... wtf happened?

This. I had one at work fail on me after 10 months of usage; got almost everything back from backups, but the timing was incredibly bad and I had to work some really late nights to pull through. Got a free replacement, but I'm pretty distrusting of it so now it's in Raid1 with a WD drive. That may've been a decent one though, because it's been working for two years now.

Re: SSDs: A gift and a curse

#83

Post by Dylan16807 »

wtallis wrote: post #40
bduerst wrote: post #36

Earlier quoted context omitted.

Doesn't that require partitioning the disk? That doesn't seem like an OOB solution.

It's not the requirement to partition the drive that kills the idea. Setting aside the first small chunk of the drive for the firmware to read is how it's always been done. The problem is that this would require the partitioning to be done below the wear leveling layer, which reduces the effectiveness of the wear leveling slightly and means you can only change your bootloader settings a thousand times before the driv…

That's not really what kills it. You can use a small ring buffer to boost that to 100 thousand, and/or use the flash in a more reliable way there (like SLC or devoting 2/3 of the bits to ECC).

Re: SSDs: A gift and a curse

#84

Post by simplexion »

insaneirish wrote: post #78

Moral of the story: * Don't use hardware RAID controllers. * Don't buy hardware from people who are going to change SKUs out from under you, or worse, change what's actually delivered for a given SKU.

This is exactly what I took from the article "Don't use hardware RAID controllers."

Re: SSDs: A gift and a curse

#85

Post by baruch »

wtallis wrote: post #46
Lagged2Death wrote: post #41

On the upside, [the ridiculously expensive HP SSDs] do have fancy detailed stats (like wear levelling) exposed via the controller and ILO, and none have failed yet almost 3 years on (in fact, they’re all showing 99% health). You get what you pay for, luckily. Call me a huge cynic if you must, but given the other problems observed, I think there's a really simple explanation for perfectly uniform "99% health" after th…

A SSD can be 85% of the way through its lifespan and still be operating with the same performance and reliability expectations it had when it was almost new, so the drive can reasonably be said to still be healthy. It's only when it has to start retiring bad blocks and expending the spare area that the drive is operating in a degraded mode, and it's not until you hit that point that the drive can start making accurat…

Not quite.

An SSD at 85% wearout will take longer to program and erase the blocks and it will require more refreshes (data retention suffers) so there will be considerably more background operations.

In fact SSD vendors do quite a bit to make the disk perform its best early on when they absolutely can and then start to break down further down the road after the benchmark is over. After all the users are mostly only testing it for a month or two and they never get to a real wearout condition during the test and then they buy in bulk and run it for a few years but by that time the SSD model is already outdated anyway (about 18 month cycle for SSD models).

Re: SSDs: A gift and a curse

#86

Post by pjc50 »

theatrus2 wrote: post #54
zanny wrote: post #50

Earlier quoted context omitted.

Sounds almost like having free software SSD firmware would be a nice thing. I just love how I have microcontrollers on all my persistent storage devices now that operate as a black box made by incompetent or possibly malicious developers.

You've always had this.

Not back in the days of loading programs from cassette tape I didn't.

Re: SSDs: A gift and a curse

#87

Post by willejs »

insaneirish wrote: post #78

Moral of the story: * Don't use hardware RAID controllers. * Don't buy hardware from people who are going to change SKUs out from under you, or worse, change what's actually delivered for a given SKU.

Totally agree with you. I was just about to write the exact same thing. I've had problems with dell hardware, hardware raid controllers, and hundreds of ssds. The most stable systems I've worked on use lsi sas controllers, perhaps software raid, and Intel/crucial ssds in fault tolerant distributed systems. Therefore, I also question if it's etsy's tendency to use non distributed datastores, with less effective or complex fault tolerance makes ssd failure a bigger deal? Failure is inevitable, and should be expected at every level.

Re: SSDs: A gift and a curse

#88

Post by linker3000 »

deelowe wrote: post #66

Isn't running raid1/5/6 on ssds silly b/c they'll all die at the same time? And hardware raid on top of that? Why? SSDs have a fairly consistent failure curve (exusing firmware bugs and other random events) for a given model, so they'll wear evenly in a raid setup. This means they'll all die at the same time as writes/reads are distributed fairly evenly across the disks. Given the size of today's drives, you may not…

In my experience (Flash Platforms Group, HSGT), a significant number of flash device failures are caused by mechanical issues, such as: on-die faults, wire bonding problems and solder joint/package stresses which may occasinally be down to a production issue, but are more often attributed to rough handling during installation or thermal stress. This is less of an issue with SFF SSDs, but especially true of PCIe products crammed into 2U servers with poor airflow.

In general, thermals tend to be a significant issue for all form factors when devices are retrofitted. Less so for 'products' (All flash arrays etc.) which are designed as whole products from the outset.

This creates a failure pattern that is totally separate from predictable wear due to use and means that 'they'll all die at the same time' becomes much less certain for some categories of device.

Re: SSDs: A gift and a curse

#89

Post by lewisl9029 »

mrb wrote: post #10

Of course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash chips on a PCB, take the controller's manufacturer's reference firmware implementation, tweak the dozens of knobs provided by the reference implementation (ignore FLUSH commands, change the amount of reserved sectors, disable this, enable that, etc), change th…

It would be interesting if SSD firmware were all open-sourced.

Manufacturers could then eventually converge all of their development and QA efforts on a few different codebases designed for different tradeoffs rather than everybody rolling their own thing and learning everybody else's lessons the hard way.

But I guess as long as custom firmware remains a major competitive advantage, this would never happen.

Re: SSDs: A gift and a curse

#90

Post by aguki »

noobie wrote: post #43
mrb wrote: post #10

Of course SSD firmware is buggy. You know why? Because any half-decent electrical and computer engineering team can slap a NAND flash controller and some flash chips on a PCB, take the controller's manufacturer's reference firmware implementation, tweak the dozens of knobs provided by the reference implementation (ignore FLUSH commands, change the amount of reserved sectors, disable this, enable that, etc), change th…

Excuse me but most of what you typed went right over my head. Where can one start learning about SSD/HDD firmware and most of what you said?

The OpenSSD project[1] might be worth a look. If you don't fancy paying thousands of dollars, you can buy SSD controller boards and raw NAND flash chips off Taobao and assemble them together. There's plenty of information on various Chinese forums (Mydigit, Upan etc.), admittedly more towards USB flash drives. Some older flash chips are TSOP rather than BGA which makes hand soldering much easier.

[1] http://www.openssd-project.org/wiki/The_OpenSSD_Project

Post reply on HN