What every programmer should know about SSDs • Hacker News Board

Re: What every programmer should know about SSDs

#121

Post by adrian_b » Mon, Jun 21, 2021, 5:33 AM UTC

crazygringo wrote: post #62

teddyh wrote: post #36

Earlier quoted context omitted.

Depending on manufacturer, and storage conditions, it can be up to about ten years. But the “three months” number is real: https://web.archive.org/web/20210502042514/http://www.dell.c...

That's a document from nine and a half years ago , and it states: > It depends on the how much the flash has been used (P/E cycle used), type of flash, and storage temperature. In MLC and SLC, this can be as low as 3 months and best case can be more than 10 years. The retention is highly dependent on temperature and workload. Are there any modern sources provide more accurate stats? "3 months to 10 years" is so vague…

Consumer SSDs (unlike enterprise SSDs) must have a retention time of at least 1 year at the end of their life.

To achieve that target, when they are new they must have a retention time of a few years, but you should better not count on that.

Re: What every programmer should know about SSDs

#122

Post by wtallis » Mon, Jun 21, 2021, 5:50 AM UTC

riobard wrote: post #110

rdc12 wrote: post #105

Earlier quoted context omitted.

The SSD maintains a translation table for all the virtual addresses exposed by the drive, that maps to the underlying flash physical addresses. Any physical address not in that table, is unallocated and the drive can use freely.

So over-provisioning has to be done before any writes to the drive? What if I want to over-provision a used drive? Discard all blocks first?

With most SSDs, there's no special explicit step necessary to overprovision a device. Just trim/unmap/discard a range of logical block addresses, and then never touch them again. The drive won't have any live data to preserve for those LBAs after they've been wiped by the trip operation, and the total amount of live data it is tracking will stay well below the advertised capacity of the drive.

The easiest way to achieve this is to create a partition with no filesystem, and use blkdiscard or similar to trim the LBAs corresponding to that partition.

Re: What every programmer should know about SSDs

#123

Post by polskibus » Mon, Jun 21, 2021, 5:51 AM UTC

natmaka wrote: post #116

naikrovek wrote: post #85

Earlier quoted context omitted.

at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit. On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE. It also is useful for a blockchain,…

Could you relate your day experience to 2ndquandrant's (contradictory?) advice?

https://www.2ndquadrant.com/en/blog/postgresql-no-tablespace...

Re: What every programmer should know about SSDs

#124

Post by Dylan16807 » Mon, Jun 21, 2021, 6:09 AM UTC

wtallis wrote: post #93

Dylan16807 wrote: post #83

Earlier quoted context omitted.

> The PS5 does hardware decompression, so games by default are now going to be compressed. If that really is cause and effect, that's a bit disappointing. For any game that isn't assuming you have an ultra-fast SSD, normal CPU decompression can handle things quite well. Such a hard nudge shouldn't have been necessary.

With few exceptions, video games have been keeping their assets on disk in compressed form for a long time. It's a major embarrassment when someone ships a game with uncompressed audio, and impractical to ship with uncompressed image, texture or video assets (though these can be shipped in compressed form with unnecessarily high resolution). The hardware decompression acceleration in new consoles doesn't exactly make…

> With few exceptions, video games have been keeping their assets on disk in compressed form for a long time.

Well, we can point to fortnite up there, but also a very large fraction of the games I have on steam can be shrunk by a third just by applying filesystem-level compression, despite it using weak algorithms and small blocks. I'm sure there's compression involved, but it's not even meeting a minimum bar of competency.

Re: What every programmer should know about SSDs

#125

Post by formerly_proven » Mon, Jun 21, 2021, 6:18 AM UTC

jedberg wrote: post #82

gruez wrote: post #81

Earlier quoted context omitted.

The kernel can't optimize that because sqlite is specifically requesting it to force a write.

Yes but you can configure the kernel to ignore that, and by default it does. For example, way back in the day, to get more life out of my laptop during college, I configured the kernel to only write to disk once an hour or when the buffer filled up. That effectively meant I was only writing to disk once per hour when I shut down to change classes. The modern linux kernel doesn't actually write to disk when fsync is c…

> The modern linux kernel doesn't actually write to disk when fsync is called. It buffers the writes in a cache.

That's not true, you can tell in many ways but one of the easiest is because fsync is quite slow and noisy (on hard drives).

Re: What every programmer should know about SSDs

#126

Post by josefx » Mon, Jun 21, 2021, 6:23 AM UTC

lazide wrote: post #111

quotemstr wrote: post #108

Earlier quoted context omitted.

Your concerns are all theoretical and the management disadvantages of direct partition access are real with or without LVM (which itself is exactly the sort of middle layer you claim to be worried about.) Do you have numbers or not?

Ah, but now you’re moving the goalposts it seems? Since most of what we’re talking about is unnecessary complexity for no real gain, what concrete metric do you think would be useful exactly? I just pointed out that you can get the same management advantages without it (say for a dev environment or rollbacks or whatever). And you get a simpler, cleaner story without extra layers if you don’t want to use lvm (such as…

The problem is some of the alternatives seem to be suggested by way of "if we had any support for this it would be better than O_DIRECT". So don't use O_DIRECT, use the alternative which doesn't exist, is still too slow, only covers parts of what you need, etc. .

Re: What every programmer should know about SSDs

#127

Post by formerly_proven » Mon, Jun 21, 2021, 6:25 AM UTC

cbsmith wrote: post #39

kortilla wrote: post #18

The title should be “why SSDs mean programmers no longer have to think about hard drives”. These are all reasons SSDs are much more pleasant to work with than old platter disks.

Well, they no longer need to think about hard disks , but there are a lot assumptions from the world of hard disks that play out very differently in the SSD world.

I don't think there's any optimization for hard drives that is going to hurt on SSDs, and unoptimized workloads are always going to work better on SSDs. I'm inclined to agree with GP that SSDs are quite close to random-access storage and so there is little to worry about.

Re: What every programmer should know about SSDs

#128

Post by raffraffraff » Mon, Jun 21, 2021, 6:39 AM UTC

natmaka wrote: post #116

naikrovek wrote: post #85

Earlier quoted context omitted.

at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit. On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE. It also is useful for a blockchain,…

Isn't this extremely dangerous? Disk write caches aren't used most of the time, except on battery backed HBAs. And databases are typically configured to use O_DIRECT for a reason: COMMITs are supposed to be durable. We had this fight at a previous company when an engineer based database server hardware recommendation on a dangerously misconfigured database server, and did not consider the effect of caches. As soon as a safe configuration was used in production, performance dropped off a cliff, particularly on random IO. So the question we had to ask was: do you want to trade durability for performance? Or do you now have to carve up your databases into shards that fit the IO performance characteristics of the badly chosen servers you purchased, and waste rack space and CPU power?

Re: What every programmer should know about SSDs

#129

Post by cbsmith » Mon, Jun 21, 2021, 6:52 AM UTC

wtallis wrote: post #72

cbsmith wrote: post #41

Earlier quoted context omitted.

...and all that cruft, and the logic to try to make handling of it not so bad, makes for a lot of complexity and unintended consequences.

Emulating 4kB or 512B sectors when the underlying media has a 16kB native page size really doesn't add much more complexity on top of the stuff that was already required to handle the fact that erase blocks are multiple megabytes.

The complexity doesn't come from the emulation. It comes from trying to do the emulation efficiently based on assumptions about the behaviour of the other moving parts... which are also doing the same thing.

So, you've got firmware that is pretending you've got 512B/4kB chunks when really you have 16kB, and anticipating how the other layers might be doing things in order to maximize performance.

Then you have a filesystem/VFS layer, which tries to optimize its access patterns anticipating how the underlying solid state storage might be really doing things in 16kB sizes and how it might be optimizing 512KB & 4kB accesses to fit that.

Both those layers are dealing with filesystem journaling and how that might impact performance.

Then you might have a database, which is now trying to anticipate how the filesystem and the underlying firmware might be optimizing access patterns, and so it's trying to optimize to fit all that.

You also potentially have application logic that is trying to anticipate how the database might do things...

What you tend to end up with are many layers of redundant caching that are all working against each other in a very inefficient manner.

Re: What every programmer should know about SSDs

#130

Post by cbsmith » Mon, Jun 21, 2021, 7:03 AM UTC

formerly_proven wrote: post #127

cbsmith wrote: post #39

Earlier quoted context omitted.

Well, they no longer need to think about hard disks , but there are a lot assumptions from the world of hard disks that play out very differently in the SSD world.

I don't think there's any optimization for hard drives that is going to hurt on SSDs, and unoptimized workloads are always going to work better on SSDs. I'm inclined to agree with GP that SSDs are quite close to random-access storage and so there is little to worry about.

Sure there are. If nothing else, hard disks have much more consistent latency characteristics for reads and writes. So, for example, you might trade some extra write IOs to ensure data is organized efficiently on disk, reducing the number of read IOs you will subsequently have. With an SSD it's largely a waste of time, because the random reads are so much cheaper and the "contiguous" blocks you think you are seeing are mapped all over the drive anyway. You want to organize things reasonably efficiently when you write, and then rewrite as little as possible, ideally never. LSM's tend to fit the SSD paradigm so much better than say... balanced trees for this reason. Similar story with clustered indexes in databases. If you use a clustered index on an SSD, usually it's for an index on something like time where new records are invariably going to go near the end of the index; anything else will have bad write performance on a hard disk, but it might be worth it for the read performance... with the SSD, it is just an unmitigated disaster.

There was a time where people thought of hard drives as "just random access storage" and consequently "there is little to worry about" and "unoptimized workloads are always going to work better on SSDs". Yup, SSDs are way faster than what came before them, but that if anything tends to mean that data structures & algorithms that used to make sense might not make much sense any more.