Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every programmer should know about SSDs

databasearchitects.blogspot.com

Post reply on HN
Print view View original

41–50 of 163 posts

Re: What every programmer should know about SSDs

#41

Post by cbsmith »

wtallis wrote: post #15
1_player wrote: post #11

A lot of talk about pages, but no mention about how big these pages are. From a quick look on Google, most SSDs have 4kB pages, with some reaching 8kB or even 16kB.

SSDs mostly tell the host system that they have 512-byte sectors or sometimes 4kB sectors, and the typical flash translation layer works in 4kB sectors because that's a good fit for the kind of workloads coming from a host system that usually prefers to do things (eg. virtual memory) in 4kB chunks. But the underlying NAND flash page size has been 16kB for years.

...and all that cruft, and the logic to try to make handling of it not so bad, makes for a lot of complexity and unintended consequences.

Re: What every programmer should know about SSDs

#43

Post by maccard »

Mikealcl wrote: post #37
maccard wrote: post #27

Earlier quoted context omitted.

This is very very true. The PS5 does hardware decompression, so games by default are now going to be compressed. For a real world reference of how big a difference that makes, see fortnite turning on compression [0] (disclaimer: I worked for epic on fortnite at the time) [0] https://www.ign.com/articles/fortnites-latest-patch-makes-it...

I had not loaded ign without blockers in years. That was painful.

Hah, sorry. Ive always found their articles to have the least fluff on the topic, despite the awful awful website.

Re: What every programmer should know about SSDs

#44

Post by pclmulqdq »

bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

I used to think the same thing, but now that I work on SSD-based storage systems, I'm not sure this holds up in today's storage stacks. Log structuring really helped with HDDs since it meant fewer seeks.

In particular, the filesystem tends to undo a lot of the benefits you get from log-structuring unless you are using a filesystem designed to keep your files log-structured. Using huge writes definitely still helps, though.

A paper that I really like goes deeper into this: http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf

Edit: I had originally said "designed for flash" instead of "designed to keep your files log-structured." F2FS is designed for flash, but in my testing does relatively poorly with log-structured files because of how it works internally.

Edit 2: de-googled the link. Thank you for pointing that out.

Re: What every programmer should know about SSDs

#45

Post by trulyme »

pclmulqdq wrote: post #44
bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

I used to think the same thing, but now that I work on SSD-based storage systems, I'm not sure this holds up in today's storage stacks. Log structuring really helped with HDDs since it meant fewer seeks. In particular, the filesystem tends to undo a lot of the benefits you get from log-structuring unless you are using a filesystem designed to keep your files log-structured. Using huge writes definitely still helps, t…

Degoogled link: http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf

Re: What every programmer should know about SSDs

#46

Post by 1-6 »

rabuse wrote: post #34

A little off topic, but I bought a new Macbook Pro with the M1 chip with 8GB of RAM, and I'm worried about the swap usage of this machine wearing out the SSD too quickly. Is this an actual concern, as my swap has been in the multiple GB range with my use?

From what I’ve been able to gather, the excessive paging may actually have to do with non-native apps running on the M1. Avoid those.

Re: What every programmer should know about SSDs

#47

Post by klodolph »

If you care about SSDs, one paper you should read is “Don’t Stack Your Log on My Log” by Yang et al. 2014

https://www.usenix.org/system/files/conference/inflow14/infl...

> Log-structured applications and file systems have been used to achieve high write throughput by sequentializing writes. Flash-based storage systems, due to flash memory’s out-of-place update characteristic, have also relied on log-structured approaches. Our work investigates the impacts to performance and endurance in flash when multiple layers of log-structured applications and file systems are layered on top of a log-structured flash device. We show that multiple log layers affects sequentiality and increases write pressure to flash devices through randomization of workloads, unaligned segment sizes, and uncoordinated multi-log garbage collection. All of these effects can combine to negate the intended positive affects of using a log. In this paper we characterize the interactions between multiple levels of independent logs, identify issues that must be considered, and describe design choices to mitigate negative behaviors in multi-log configurations.

Re: What every programmer should know about SSDs

#48

Post by 10000truths »

pclmulqdq wrote: post #44
bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

I used to think the same thing, but now that I work on SSD-based storage systems, I'm not sure this holds up in today's storage stacks. Log structuring really helped with HDDs since it meant fewer seeks. In particular, the filesystem tends to undo a lot of the benefits you get from log-structuring unless you are using a filesystem designed to keep your files log-structured. Using huge writes definitely still helps, t…

Achieving cutting-edge storage performance tends to require bypassing the filesystem anyways. Traditionally, that meant using SPDK. Nowadays, opening /dev/nvme* with O_DIRECT and operating on it with io_uring will get you most of the way there.

In either case, the advice given in the article and by the OP is filesystem agnostic.

Re: What every programmer should know about SSDs

#49

Post by remram »

bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

Shouldn't the OS or libc take care of that? If I write and don't immediately flush()?

Re: What every programmer should know about SSDs

#50

Post by rossdavidh »

Interesting, and fun to read and think about! And, as a professional programmer for 17 years now, not once have I done anything where this would have been important for me to know (even if I had been running my code on a system with SSD's). So, I'm not convinced the title is at all accurate.

But, fun to read and think about.

Post reply on HN