Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every programmer should know about SSDs

databasearchitects.blogspot.com

Post reply on HN
Print view View original

111–120 of 163 posts

Re: What every programmer should know about SSDs

#111

Post by lazide »

quotemstr wrote: post #108
lazide wrote: post #84

Earlier quoted context omitted.

You do when it does that/respects it which isn’t always. The point is that you have more layers. If you’re trying to be as direct as possible, more layers is unhelpful. Since you get most of the same advantages management wise with lvm while using the block interface (including snapshots, resizing, and all the other management goodies), you’re not exactly getting much extra functionality either.

Your concerns are all theoretical and the management disadvantages of direct partition access are real with or without LVM (which itself is exactly the sort of middle layer you claim to be worried about.) Do you have numbers or not?

Ah, but now you’re moving the goalposts it seems?

Since most of what we’re talking about is unnecessary complexity for no real gain, what concrete metric do you think would be useful exactly? I just pointed out that you can get the same management advantages without it (say for a dev environment or rollbacks or whatever). And you get a simpler, cleaner story without extra layers if you don’t want to use lvm (such as in production), which you can’t get from O_DIRECT.

I also have this thread from Linus calling O_DIRECT brain damaged and to never use it. [https://lkml.org/lkml/2007/1/10/233]

Re: What every programmer should know about SSDs

#112

Post by wly_cdgr »

hddherman wrote: post #109
wly_cdgr wrote: post #66

There's nothing whatsoever I should need to know about SSDs as a Javascript programmer and if there is then the programmers on the lower levels haven't done their jobs right and are wasting my time

Ever heard of leaky abstractions?

Sure, yeah...that's the "haven't done their jobs right and are wasting my time" part

Re: What every programmer should know about SSDs

#113

Post by riobard »

ars wrote: post #101
riobard wrote: post #100

One thing I'm still puzzled about SSD over-provisioning, which is also mentioned by the tutorial ( https://codecapsule.com/2014/02/12/coding-for-ssds-part-4-ad... ) recommended by the article: > A drive can be over-provisioned simply by formatting it to a logical partition capacity smaller than the maximum physical capacity. The remaining space, invisible to the user, will still be visible and used by the SSD control…

Any sector with nothing written on it can be used as scrap. So if you partition the entire thing, but just never write to the full disk (you never use all the space), that also works as overprovisioning. Partitioning just forces that to happen.

If I partition the entire drive, eventually all blocks will be used, depending on how the filesystem allocates, right? So to guarantee some free space it's better to over-provision by under-partitioning. Now how do I make sure that on a used drive?

Re: What every programmer should know about SSDs

#115

Post by ars »

riobard wrote: post #113
ars wrote: post #101

Earlier quoted context omitted.

Any sector with nothing written on it can be used as scrap. So if you partition the entire thing, but just never write to the full disk (you never use all the space), that also works as overprovisioning. Partitioning just forces that to happen.

If I partition the entire drive, eventually all blocks will be used, depending on how the filesystem allocates, right? So to guarantee some free space it's better to over-provision by under-partitioning. Now how do I make sure that on a used drive?

That's what the trim command does.

It runs periodically and lets the SSD know about unused areas.

So as long as you don't fill up the drive and let trim do its thing the unused areas effectively do the same thing as over provisioning.

Re: What every programmer should know about SSDs

#116

Post by natmaka »

naikrovek wrote: post #85
10000truths wrote: post #69

Earlier quoted context omitted.

Extreme performance requires extreme tradeoffs. As with anything else, you have to evaluate your use cases and determine for yourself whether the tradeoffs are worth it. For a mass-market application that has to play nice with other applications and work with a wide variety of commodity hardware, it's probably not worthwhile. For a state-of-the-art high performance data store that expects low latencies and high throu…

at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit.

On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE.

It also is useful for a blockchain, amazingly fast (and a relief for HDDs), in most cases alleviating the need for a SSD. Place the blockchain file(s) on the tmpfs mount. Before machine shutdown stop any blockchain-using software, then store a compressed copy of the blockchain file(s) on permanent storage (I use "zstd -T0 --fast"...), and upon reboot restore it on the tmpfs mount. If anything fails the blockchain-writing software will re-download any missing block.

Re: What every programmer should know about SSDs

#117

Post by fulafel »

ww520 wrote: post #106
bob1029 wrote: post #26

Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your…

LMDB has similar write characteristics where its b-tree is append-only. This gives LMDB amazing performance and very robust ACID transaction support as immutability is baked in.

This is quite common in traditional DBs too. Eg PostgreSQL has its write-ahead log. Both LMDB and PostgreSQL then occasionally need to do do some kind of compaction, checkpoint or garbage collection, whatever it's called in various systems, the write-only log is reset and any live data in it improted into the main db data.

Re: What every programmer should know about SSDs

#118

Post by ksec »

rabuse wrote: post #34

A little off topic, but I bought a new Macbook Pro with the M1 chip with 8GB of RAM, and I'm worried about the swap usage of this machine wearing out the SSD too quickly. Is this an actual concern, as my swap has been in the multiple GB range with my use?

Generally speaking macOS is extremely write heavy for all sort of reason even before the switch to ARM. But in majority of case if should last 4-5 years without problem.

The heavy write bug Apple said was due to misreporting and was fixed ( so they say ).

I do think you should pay attention to it from time to time. iCloud Sync, Spotlight, Safari heavy tabs are all known to cause heavy paging in some corner case. You might end up having a TB of data written for no apparent reason. Apple used to ship their Macbook with MLC, on a 512GB MLC you could do 500TBW without problem, that is ~13 years of usage if you do 100GB write per day. Not sure about the M1 machines.

If you are doing Dev staging, Video and photos editing a lot these drive will fail quite quickly. In the space of 2 - 3 years. Although some would argue MacBook Air are not made for those task. And especially true if you have 8GB and 256GB NAND.

Re: What every programmer should know about SSDs

#119

Post by fulafel »

jedberg wrote: post #16

This page tells me a lot about SSDs, but it doesn't tell me why I need to know these things. It doesn't really give me any indication about how I should change my behavior if I know that I'll be running on SSD vs spinning disk. I've always been told, "just treat SSDs like slow, permanent memory".

Indeed. The summary talks about what you need to do to saturate a SSDs read and write bandwidth. I guess the post would find its audience better if the title was "What a programmer should about SSDs when optimizing IO".

I'd be more interested in the trends in SSD behaviour are. It seems SSDs have bigger and bigger DRAM caches and wear ceased to be an issue many years ago, so there's not much payoff in the write side advice of the article.

Re: What every programmer should know about SSDs

#120

Post by adrian_b »

natmaka wrote: post #116
naikrovek wrote: post #85

Earlier quoted context omitted.

at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit. On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE. It also is useful for a blockchain,…

While tmpfs can be very useful even as it is, users must beware that copying a file from another Linux file system to tmpfs can lose a part of the file metadata, without giving any warnings or errors.

The main problem is that copying a file to tmpfs will drop extended attributes. Old versions of tmpfs dropped all extended attributes, modern versions of tmpfs keep some security-related extended attributes, but they still drop any user-defined extended attributes.

Old versions of tmpfs truncated some high-resolution timestamps, e.g. those coming from xfs, but I do not know if this still happens on modern versions of tmpfs.

Before learning these facts, I could not understand while some file copies lost parts of their metadata, after being copied via /tmp between 2 different users, on a multi-user computer where /tmp was mounted on tmpfs.

Now that I know, when I have to copy a file via tmpfs, I have to make a pax archive, which preserves file metadata. Older tar archive formats may have the same problems like tmpfs.

Post reply on HN