Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every programmer should know about SSDs

databasearchitects.blogspot.com

Post reply on HN
Print view View original

91–100 of 163 posts

Re: What every programmer should know about SSDs

#91

Post by wtallis »

gruez wrote: post #80
ifdefdebug wrote: post #75

Earlier quoted context omitted.

For instance, when reading this sqlite came immediately to my mind and how much a 10000 loop of inserts without begin/commit or some preparing pragmas would wreck a ssd... (forces a full sync between each two inserts)

Fortunately most people aren't running OLTP workloads on client SSDs. That's mostly done on enterprise SSDs that have much higher endurance. That said even on client SSDs you can probably get away with running such workloads as long as you're not doing them 24/7.

More important than the higher rated endurance (and perhaps contributing a bit to that rating) is the fact that the typical enterprise SSD has power loss protection capacitors for its RAM, so it can cache and combine writes in RAM safely.

Re: What every programmer should know about SSDs

#92

Post by 10000truths »

naikrovek wrote: post #85
10000truths wrote: post #69

Earlier quoted context omitted.

Extreme performance requires extreme tradeoffs. As with anything else, you have to evaluate your use cases and determine for yourself whether the tradeoffs are worth it. For a mass-market application that has to play nice with other applications and work with a wide variety of commodity hardware, it's probably not worthwhile. For a state-of-the-art high performance data store that expects low latencies and high throu…

at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

> at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs.

A storage application that need to bypass the filesystem will already be implementing its own caching system anyways. The idea is to persist the data to maintain durability without sacrificing latency.

> manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

That is what O_SYNC flag is for.

Re: What every programmer should know about SSDs

#93

Post by wtallis »

Dylan16807 wrote: post #83
maccard wrote: post #27

Earlier quoted context omitted.

This is very very true. The PS5 does hardware decompression, so games by default are now going to be compressed. For a real world reference of how big a difference that makes, see fortnite turning on compression [0] (disclaimer: I worked for epic on fortnite at the time) [0] https://www.ign.com/articles/fortnites-latest-patch-makes-it...

> The PS5 does hardware decompression, so games by default are now going to be compressed. If that really is cause and effect, that's a bit disappointing. For any game that isn't assuming you have an ultra-fast SSD, normal CPU decompression can handle things quite well. Such a hard nudge shouldn't have been necessary.

With few exceptions, video games have been keeping their assets on disk in compressed form for a long time. It's a major embarrassment when someone ships a game with uncompressed audio, and impractical to ship with uncompressed image, texture or video assets (though these can be shipped in compressed form with unnecessarily high resolution).

The hardware decompression acceleration in new consoles doesn't exactly make it easier to use compression for the game assets. Rather, it makes it practical to load compressed assets on-demand instead of reading and decompressing into RAM during the loading screen.

Re: What every programmer should know about SSDs

#94

Post by KMag »

remram wrote: post #49
bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

Shouldn't the OS or libc take care of that? If I write and don't immediately flush()?

I don't think most libc implementations take care to buffer to filesystem block/cluster boundaries.

Re: What every programmer should know about SSDs

#95

Post by wtallis »

ropeladder wrote: post #87

If sequential and random reads are mostly the same on SSDs, does that make the distinction between columnar and row-based databases/data storage less important?

Nope, unless your columns are all several kB wide. If you force the hardware to perform a multi-kB read for each 64-bit value you need, you're still going to waste a lot of potential performance.

Re: What every programmer should know about SSDs

#96

Post by gjs278 »

jedberg wrote: post #16

This page tells me a lot about SSDs, but it doesn't tell me why I need to know these things. It doesn't really give me any indication about how I should change my behavior if I know that I'll be running on SSD vs spinning disk. I've always been told, "just treat SSDs like slow, permanent memory".

the former sysadmin of reddit when it was constantly offline doesn’t understand ssds better than slow, permanent memory. it’s all starting to add up.

Re: What every programmer should know about SSDs

#97

Post by cottsak »

dang wrote: post #3

What someone else said about that in 2014: What every programmer should know about solid-state drives - https://news.ycombinator.com/item?id=9049630 - Feb 2015 (31 comments)

haha! very similar sections too .. almost looked copied for a brief moment as i skimmed there

Re: What every programmer should know about SSDs

#98

Post by nyanpasu64 »

10000truths wrote: post #69
nyanpasu64 wrote: post #59

Earlier quoted context omitted.

Will an end user downloading a video editing app (or similar) have a NVME drive, know how to give your app direct access to a NVME drive, and will your app not corrupt the rest of the files on the drive?

Extreme performance requires extreme tradeoffs. As with anything else, you have to evaluate your use cases and determine for yourself whether the tradeoffs are worth it. For a mass-market application that has to play nice with other applications and work with a wide variety of commodity hardware, it's probably not worthwhile. For a state-of-the-art high performance data store that expects low latencies and high throu…

Would high-performance data storage be easier to implement on commodity hardware if operating systems supplied an API to get a blob of bytes, segmented out of an entire disk (eg. a file), that presented low-level semantics like a full-fledged SSD partition or drive?

I feel that operating systems need to provide self-contained reliable APIs designed for atomically overwriting configuration files, without losing permissions or overwriting symlinks or such. Or perhaps supply more powerful primitives, like a faster/weaker fsync that serves as an ordering barrier rather than flushing to disk, or an API to replace a file without altering permissions. One issue I've heard is:

> I even had an issue with atomic writes over ssh that created the temp file but where not able to rename it, so the old one stayed.

Re: What every programmer should know about SSDs

#99

Post by cottsak »

rossdavidh wrote: post #50

Interesting, and fun to read and think about! And, as a professional programmer for 17 years now, not once have I done anything where this would have been important for me to know (even if I had been running my code on a system with SSD's). So, I'm not convinced the title is at all accurate. But, fun to read and think about.

I think the key is hidden in > which can help creating software that is capable of exploiting them

Unless you're writing desktop software or your application behaves in a way where you have actually selected the particular hardware components (most of us in cloud hosting don't do this), you probably don't [need to] care.

Re: What every programmer should know about SSDs

#100

Post by riobard »

One thing I'm still puzzled about SSD over-provisioning, which is also mentioned by the tutorial (https://codecapsule.com/2014/02/12/coding-for-ssds-part-4-ad...) recommended by the article:

> A drive can be over-provisioned simply by formatting it to a logical partition capacity smaller than the maximum physical capacity. The remaining space, invisible to the user, will still be visible and used by the SSD controller.

Does the controller read the partition table to decide that the space beyond logic partition is safe to use as scrap?

Post reply on HN