What we learned about SSDs in 2015 • Hacker News Board

Re: What we learned about SSDs in 2015

#31

Post by wtallis » Sun, Dec 20, 2015, 7:09 PM UTC

nly wrote: post #21

pkaye wrote: post #5

Earlier quoted context omitted.

NVMe is just the communications protocol between the host and device. Garbage collection is due to hiding the non ideal properties of the NAND from the host. Primary among them is endurance limits and the size difference between write and erase units. You could for example move the handling of some of these GC details to the OS filesystem level but then that becomes more complicated and has to deal with the differenc…

> You could for example move the handling of some of these GC details to the OS filesystem level Isn't this basically the Linux 'discard' mount flag? Most distros seem to be recommending periodic fstrim for consumer uses, what's the best practice in the data center?

Discard operations only tell the drive that a LBA is eligible for GC. It's basically a delete operation that explicitly can be deferred. It does not give the OS any input into when or how GC is done, and it doesn't give the OS any way to observe any details about the GC process.

I think the recommendations for periodic fstrim of free space is due to filesystems usually not taking the time to issue a large number of discard operations when you delete a bunch of data. Even though discards should be faster than a synchronous erase command, not issuing any command to the drive is faster still.

Re: What we learned about SSDs in 2015

#32

Post by pkaye » Sun, Dec 20, 2015, 7:10 PM UTC

nly wrote: post #21

pkaye wrote: post #5

Earlier quoted context omitted.

NVMe is just the communications protocol between the host and device. Garbage collection is due to hiding the non ideal properties of the NAND from the host. Primary among them is endurance limits and the size difference between write and erase units. You could for example move the handling of some of these GC details to the OS filesystem level but then that becomes more complicated and has to deal with the differenc…

> You could for example move the handling of some of these GC details to the OS filesystem level Isn't this basically the Linux 'discard' mount flag? Most distros seem to be recommending periodic fstrim for consumer uses, what's the best practice in the data center?

The trim is kind of like the free counterpart to malloc. When the drive has been fully written, there is a lot less free space which constraints the GC and makes it works much harder. The trim tells the drive to free up space allowing for less GC work.

A big difference between client and enterprise drives is the amount of over-provisioning. A simple trick if you can't use trim is to leave 5-10% of the drive unused to improve the effective over-provisioning and improve worst case performance.

The question of using trim in data centers might be due to interaction between trim and raid configurations because sometimes trim is implemented as nondeterministic (as a hint to the drive; persistant trim can cause performance drops unless you are willing to through in extra hardware to optimize it well) which causes parity calculation issues when recovering from a failed drive.

Re: What we learned about SSDs in 2015

#33

Post by pkaye » Sun, Dec 20, 2015, 7:14 PM UTC

wtallis wrote: post #31

nly wrote: post #21

Earlier quoted context omitted.

> You could for example move the handling of some of these GC details to the OS filesystem level Isn't this basically the Linux 'discard' mount flag? Most distros seem to be recommending periodic fstrim for consumer uses, what's the best practice in the data center?

Discard operations only tell the drive that a LBA is eligible for GC. It's basically a delete operation that explicitly can be deferred. It does not give the OS any input into when or how GC is done, and it doesn't give the OS any way to observe any details about the GC process. I think the recommendations for periodic fstrim of free space is due to filesystems usually not taking the time to issue a large number of d…

SATA drives until recently didn't have queued trims so if you did an occasional trim between read/writes you would have to flush the queue. Queued trims were added later on but have been slow to be adopted because it can be difficult to get it working fast, efficient and correct when intermingled with reads and writes. I know atleast one recent drive with queued trim had some bugs in the implementation.

Re: What we learned about SSDs in 2015

#34

Post by scurvy » Sun, Dec 20, 2015, 7:16 PM UTC

japaw wrote: post #11

"2015 was the beginning of the end for SSDs in the data center." is quit a bold statement especial when not discussing any alternative. I do not see us going back to magnetic disk, and most new storage technology are some kind of ssd...

My thoughts exactly. The article is quite inflammatory and tosses out some bold statements without really deep diving into them. My favorite:

"Finally, the unpredictable latency of SSD-based arrays - often called all-flash arrays - is gaining mind share. The problem: if there are too many writes for an SSD to keep up with, reads have to wait for writes to complete - which can be many milliseconds. Reads taking as long as writes? That's not the performance customers think they are buying."

This is completely false in a properly designed server system. Use the deadline scheduler with SSD's so that reads aren't starved from bulk I/O operations. This is fairly common knowledge. Also, if you're throwing too much I/O load at any storage system, things are going to slow down. This should not be a surprise. SSD's are sorta magical (Artur), but they're not pure magic. They can't fix everything.

While Facebook started out with Fusion-io, they very quickly transitioned to their own home-designed and home-grown flash storage. I'd be wary of using any of their facts or findings and applying them to all flash storage. In short, these things could just be Facebook problems because they decided to go build their own.

He also talks about the "unpredictability of all flash arrays" like the fault is 100% due to the flash. In my experience, it's usually the RAID/proprietary controller doing something unpredictable and wonky. Sometimes the drive and controller do something dumb in concert, but it's usually the controller.

EDIT: It was 2-3 years ago that flash controller designers started to focus on uniform latency and performance rather than concentrating on peak performance. You can see this in the maturation of I/O latency graphs from the various Anandtech reviews.

Re: What we learned about SSDs in 2015

#35

Post by lsc » Sun, Dec 20, 2015, 7:19 PM UTC

tehwalrus wrote: post #29

tw04 wrote: post #25

Earlier quoted context omitted.

How exactly does connecting to the PCIe bus obviate the need for RAID? RAID isn't about connecting drives, it's about not losing data. If you don't need RAID for your application today, you can purchase a non-RAID SAS adapter for a couple hundred bucks, or just use the onboard ports that are sure to be there.

> RAID isn't about connecting drives, it's about not losing data. RAID-0 is used as a way to get faster performance from spinning disk drives, as you can return parts of each read request from different (striped) drives. You also get better write performance, as your writes are split across the drives.

uh, this shouldn't be voted down. Now, I don't use RAID for performance, I use raid for reliability, like parent said, and I've never actually been in a position where it would make sense, but people do use raid0 to increase performance. It happens, even if it's not nearly as common as using raid to prevent data loss.

Re: What we learned about SSDs in 2015

#36

Post by dajonker » Sun, Dec 20, 2015, 7:21 PM UTC

tehwalrus wrote: post #29

tw04 wrote: post #25

Earlier quoted context omitted.

How exactly does connecting to the PCIe bus obviate the need for RAID? RAID isn't about connecting drives, it's about not losing data. If you don't need RAID for your application today, you can purchase a non-RAID SAS adapter for a couple hundred bucks, or just use the onboard ports that are sure to be there.

> RAID isn't about connecting drives, it's about not losing data. RAID-0 is used as a way to get faster performance from spinning disk drives, as you can return parts of each read request from different (striped) drives. You also get better write performance, as your writes are split across the drives.

RAID 0 does not have any level of redundancy, so you might as well remove the 'R' in RAID and replace it by 'S' for striping or something. However, people would probably get confused if you start calling it SAID.

Re: What we learned about SSDs in 2015

#37

Post by lsc » Sun, Dec 20, 2015, 7:23 PM UTC

wtallis wrote: post #23

velox_io wrote: post #22

The most exciting recent development in SSDs (until 3DXpoint is released), is bypassing the SATA interface, connecting drives straight into the PCIe bus (no more expensive raid controllers). Just a shame hardly any server motherboards come with M.2 slots right now. The 4x speed increase and lower CPU overhead means it is now possible to move RAM only applications (for instance in-memory databases) to SSDs, keeping on…

M.2 slots don't make much sense for servers, at least if you're trying to take advantage of the performance benefits possible with a PCIe interface. Current M.2 drives aren't even close to saturating the PCIe 3.0 x4 link but they're severely thermally limited for sustained use and they're restricted in capacity due to lack of PCB area. Server SSDs should stick with the traditional half-height add-in card form factor…

most of the NVMe backplanes I've seen give full 'enterprise 2.5" drive clearance to the thing, so if they are actually as thick as consumer SSDs, as most current SATA 'enterprise SSD' are, there's plenty of room for a heatsink without expanding the slot. The supermicro chassis (and I've only explored the NVMe backplanes from supermicro) usually put a lot of effort into drawing air through the drives, so assuming you put in blanks and stuff, the airflow should be there, if the SSD are setup to take advantage of it.

Re: What we learned about SSDs in 2015

#38

Post by jjtheblunt » Sun, Dec 20, 2015, 7:30 PM UTC

ddorian43 wrote: post #16

acquacow wrote: post #12

...unless you are Fusion-io, in which case, most of these problems don't affect you.

why? isn't fusion-io based on ssd ?

fusion-io (i believe, but please verify online) uses a spinning drive for frequent writes and an ssd for frequent reads, with software deciding what goes where, lessening the write traffic to the ssd, and thus wear to it.

Re: What we learned about SSDs in 2015

#39

Post by bravura » Sun, Dec 20, 2015, 7:40 PM UTC

I need an external disk for my laptop that I leave plugged in all the time.

What is the most reliable external hard drive type? I thought SSDs were more reliable than spinning disks, especially to leave plugged in constantly, but now I'm not as sure.

Re: What we learned about SSDs in 2015

#40

Post by ak217 » Sun, Dec 20, 2015, 7:48 PM UTC

This article has a lot of good information, but its weirdly sensationalistic tone detracts from it. I appreciate learning more about 3D Xpoint and Nantero, but SSDs are not a "transitional technology" in any real sense of the word, and they won't be displaced by anything in 2016, if nothing else because it takes multiple years from volume capability to stand up a product pipeline on a new memory technology, and more years to convince the enterprise market to start deploying it. The most solid point the article makes is that the workload-specific performance of SSD-based storage is still being explored, and we need better tools for it.