Live data from Hacker News

Testing disks: Lessons from our odyssey selecting replacement SSDs

bbc.co.uk

51–60 of 78 posts

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#51
One lesson here is that when reusing a previous test setup you ought to look for assumptions you made which are no longer valid.

If they'd been starting from scratch, while thinking about modern SSDs, it's quite likely they wouldn't have built an application load tester using files containing only dots.

But as it was an existing system, it didn't get the same amount of attention.

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#52
post #32

The biggest lesson to take away from this is probably that they thought they knew how to test a SSD, but were quite obviously clueless: > we run a fairly comprehensive set of block-level tests using fio, consisting of both sequential and random asynchronous reads and writes straight to the disk. Then we throw a few timed runs of the venerable dd program at it. Running dd as a benchmark is a major red flag. It show th…

BBC published a technical (but really PR) article written by amateurs posing as pros, instead of consulting reputable experts?

To be clear, this isn't a news article written by our journalists - it is a piece written by the team themselves that we felt may be of interest to others, and that might help us do things better in the future. While I enjoyed reading it, I can assure you that SSD performance testing doesn't move the BBC PR needle compared to the identity of the new Doctor Who.

We're acutely aware that we've still got much to learn in this space, so if there are thoughts you have on how we could do better we're all ears.

Finally, while I assured you it wasn't a PR piece we're always looking for engineers in this area (and across the whole BBC) so if you'd be interested in helping us improve, get in touch.

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#53
post #46
post #45

Earlier quoted context omitted.

There is some work on Open-Channel SSDs, that move most of the flash translation layer (FTL) to the host system. There are two major problems with this approach: 1. Each OS that wants to use the drive needs a compatible implementation of the FTL. Consumer systems always have at least two operating systems in play (UEFI counts for these purposes). Enterprise systems are where you will actually find non-boot data-only…

Could you provide more details on these hints? Are they ioctl calls? Assuming one is using the disk as a raw block device, without a filesystem.

I was referring to extensions to the command set the OS uses to interact with the drive itself. Some of these are quite like a madvise() call, but at a lower layer. Others permit the drive to expose a bit more information to the OS so that it can better optimize its IO patterns. I summarized the most recently standardized changes at [1], but there are several other features in the NVMe spec [2] that fall into this category. The extension for IO determinism has been approved for the next standard but the official spec for it hasn't been published. (I'm referring here mostly to NVMe stuff, but there are SCSI/SAS analogs to many of these features.)

[1] https://www.anandtech.com/show/11436/nvme-13-specification-p...

[2] http://www.nvmexpress.org/resources/specifications/

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#54

Great article, easy to follow considering it's far away from my normal domain. I noticed they didn't mention any brands by name though, why is that?

> I noticed they didn't mention any brands by name though, why is that? As a public service broadcaster in the UK, the BBC must be very careful about naming specific brands and products due to rules laid out in their legal remit (and fear of legal action if from a party that feels unfairly disadvantaged by a competitor getting a good mention or them getting a bad one). This is way Blue Peter always use "sticky backed…

> like Google -> to google

Hang on! Surely, "to google" refers to performing a search specifically with Google, right?

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#55
post #47
post #43

Earlier quoted context omitted.

It's not just a matter of how many bytes are being written; desktop workloads rarely need to do synchronous writes.

There are a lot of desktop applications that does synchronous all the time. Chrome, Firefox and Spotify are a few examples as they use SQLite which does fsync() system call. But yeah, you will be fine even with a spinning hard drive. My point is, what use case is there for paying a premium for the Samsung NVMe's when you would be just fine with a cheaper model.

The IO done by applications like web browsers and Spotify may be synchronous from their perspective (i.e. write(2) syscalls instead of aio), but those applications definitely don't need the sort of hard ACID transactional guarantees that require all disk caches to be disabled and every single write command sent to the drive to be a barrier. The fact that they're running atop journaling filesystems is good enough.

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#56
post #50
post #30

Earlier quoted context omitted.

The 960 EVO is a consumer grade SSD with firmware tuned for bursts of I/O (through eg. the use of SLC write caching) at the expense of sustained write throughput. It doesn't have power loss protection capacitors, so it can't perform safe write caching when you're issuing synchronous writes. 4kB is much smaller than the underlying page size of its NAND flash, so performance is going to suck without write combining. Yo…

So what is the use case for this drive? The 960 Evo/Pro are supposed to be premium models, but a better investment would be a cheaper SSD drive with more storage. And if you rarely write that much, more ram will increase the read speed significantly.

a consumer pc.

the pro does not have the buffer the evo does. the evo is not a premium model, it is entry level cutting edge

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#57

Earlier quoted context omitted.

> I noticed they didn't mention any brands by name though, why is that? As a public service broadcaster in the UK, the BBC must be very careful about naming specific brands and products due to rules laid out in their legal remit (and fear of legal action if from a party that feels unfairly disadvantaged by a competitor getting a good mention or them getting a bad one). This is way Blue Peter always use "sticky backed…

> like Google -> to google Hang on! Surely, "to google" refers to performing a search specifically with Google, right?

Just like a kleenex must be a tissue produced by Kimberly-Clark or one can only xerox on equipment manufactured by a particular company? That ship is in the process of leaving the port :)

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#58

Flash memory has three operations, read, write and erase, the last two destructively. If you pretend they are harddisks with two operations of read and write you go through all sorts of contortions. Sometimes you fall flat on the face, as seen here. Why don't operating systems treat SSDs more flash memory, and why doesn't the file system cooperate with the underlying hardware instead of pretending it's a disk? For ho…

Speaking as someone who used to work on SSD firmware, here are some rambling thoughts... Yes moving some of the FTL to the OS will help a lot in reducing the complexity for the SSD developer but the problems are just moved up the levels. The OS will probably still have to use a COW scheme aware of the block and page size restrictions of the underlying flash. And you can't do a raw disk copy without accounting for defective blocks. Maybe the SSD will still handle basic ECC protection and data scrambling but the OS will now have to handle read disturb, wear leveling, defect management, and data recovery using signal processing. But many of these characteristics will change from one NAND technology to another so someone will have to characterize and update the algorithms. I would actually say it is this last bit that really trips up SSD firmware design. Otherwise you would think after an iteration or two of firmware we would have a solid design but the flash technology tends to bring up some new requirements with each node that introduces more complexity.

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#59
post #56
post #50

Earlier quoted context omitted.

So what is the use case for this drive? The 960 Evo/Pro are supposed to be premium models, but a better investment would be a cheaper SSD drive with more storage. And if you rarely write that much, more ram will increase the read speed significantly.

a consumer pc. the pro does not have the buffer the evo does. the evo is not a premium model, it is entry level cutting edge

That is not very specific, a consumer PC would be fine with a 750 Evo also, maybe two of them in raid 0 for twice the sequential read & write speed. I believe for most consumers, having more SSD storage per $ is more important.

Re: Testing disks: Lessons from our odyssey selecting replacement SSDs

#60
post #26

This is the problem IMHO > We also looked up whether our HBA used TRIM in its current configuration. It turns out, in RAID mode, the HBA did not support TRIM. We did do some trim-enabled testing with a different machine, but these results are hard to compare fairly. In any case, we can't currently enable TRIM on our production systems. In our experience SSD write performance goes to sh*t if you don't regularly TRIM t…

> In our experience SSD write performance goes to sh t if you don't regularly TRIM them. Interesting, is that because of the load? It seemed "modern" SSDs have GCs good enough that trim isn't quite necessary anymore to ensure good performances in consumer loads. > RAID cards not passing TRIM is a big problem for us too... Are there NVMe RAID cards? I assume they'd necessarily pass the command along considering deallo…

> Interesting, is that because of the load? It seemed "modern" SSDs have GCs good enough that trim isn't quite necessary anymore to ensure good performances in consumer loads.

A drive has no way to tell whether filesystem is using a given block or not. TRIM is a way for the filesystem to tell it that. So I would imagine the GC that you're referring to is working on the blocks marked with TRIM.

BTW, besides running fstrim from cron on Linux, you can also use discard flag to mount the drive, so the filesystem sends TRIM command when files are deleted.

Post reply on HN