Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every programmer should know about SSDs

databasearchitects.blogspot.com

Post reply on HN
Print view View original

141–150 of 163 posts

Re: What every programmer should know about SSDs

#141

Post by teddyh »

mercora wrote: post #30
teddyh wrote: post #19

What everyone should know is that flash drives can lose their data when left unpowered for as little as three months.

if that is true disks should come with a very visible note stating this... seriously, 3 months would be nothing. i doubt it is true because 3 months is a time frame which should be surpassed quite often making this more known.

It occurs to me now that the key word here may be “unpowered”. As in, if you unplug an SSD and leave it on the shelf, it may lose (some) data in as little as three months. There might not be many people who do that, and those who do might not notice the occasional corruption.

Re: What every programmer should know about SSDs

#142

Post by Liquid_Fire »

rabuse wrote: post #65
1-6 wrote: post #46

Earlier quoted context omitted.

From what I’ve been able to gather, the excessive paging may actually have to do with non-native apps running on the M1. Avoid those.

Most of my programs are JetBrains IDE's and browsers. Don't know if they're optimized for M1.

AFAIK most of JetBrains' IDEs are now native (other than Android Studio, which is still WIP). The mainstream browsers are also all native.

Remaining non-native apps include Dropbox, Spotify, LibreOffice and a few others. And basically all games with very few exceptions.

This website has a decently up-to-date list of what has been ported and what hasn't: https://isapplesiliconready.com/

Re: What every programmer should know about SSDs

#143

Post by barrkel »

raffraffraff wrote: post #128
natmaka wrote: post #116

Earlier quoted context omitted.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit. On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE. It also is useful for a blockchain,…

Isn't this extremely dangerous? Disk write caches aren't used most of the time, except on battery backed HBAs. And databases are typically configured to use O_DIRECT for a reason: COMMITs are supposed to be durable. We had this fight at a previous company when an engineer based database server hardware recommendation on a dangerously misconfigured database server, and did not consider the effect of caches. As soon as…

Parent is talking about temporary tables. Those are normally only live for the duration of a transaction (well, session, but in practice if you're using temporary tables across multiple transactions you have a logical application-level transaction which needs to be able to handle failure part-way through). After your transaction the writes to non-temporary tables should be persistent.

Postgres temp tables on ramdisk are a problem for a different reason, the WAL, as pointed to by a sibling comment.

Re: What every programmer should know about SSDs

#145

Post by natmaka »

barrkel wrote: post #143
raffraffraff wrote: post #128

Earlier quoted context omitted.

Isn't this extremely dangerous? Disk write caches aren't used most of the time, except on battery backed HBAs. And databases are typically configured to use O_DIRECT for a reason: COMMITs are supposed to be durable. We had this fight at a previous company when an engineer based database server hardware recommendation on a dangerously misconfigured database server, and did not consider the effect of caches. As soon as…

Parent is talking about temporary tables. Those are normally only live for the duration of a transaction (well, session, but in practice if you're using temporary tables across multiple transactions you have a logical application-level transaction which needs to be able to handle failure part-way through). After your transaction the writes to non-temporary tables should be persistent. Postgres temp tables on ramdisk…

> Postgres temp tables on ramdisk are a problem for a different reason, the WAL, as pointed to by a sibling comment

TEMPORARY tables are UNLOGGED, and therefore they aren't WALed

Re: What every programmer should know about SSDs

#146

Post by natmaka »

polskibus wrote: post #123
natmaka wrote: post #116

Earlier quoted context omitted.

Given enough RAM on a Linux machine one may use tmpfs, which maintains a RAM disk and at any moment only uses the amount of RAM needed, with a pre-defined limit. On PostgreSQL create an adequately-caped tmpfs, create a TABLESPACE on it, then store temporary tables into this TABLESPACE. No SSD (I have access to) beats this. Hint: before shutting PG down you may DROP this TABLESPACE. It also is useful for a blockchain,…

Could you relate your day experience to 2ndquandrant's (contradictory?) advice? https://www.2ndquadrant.com/en/blog/postgresql-no-tablespace...

TEMPORARY tables are UNLOGGED, and therefore they aren't WALed

See https://www.postgresql.org/message-id/CAB7nPqTkZvESuZ3qcN_Tj...

Re: What every programmer should know about SSDs

#147

Post by Unklejoe »

10000truths wrote: post #48
pclmulqdq wrote: post #44

Earlier quoted context omitted.

I used to think the same thing, but now that I work on SSD-based storage systems, I'm not sure this holds up in today's storage stacks. Log structuring really helped with HDDs since it meant fewer seeks. In particular, the filesystem tends to undo a lot of the benefits you get from log-structuring unless you are using a filesystem designed to keep your files log-structured. Using huge writes definitely still helps, t…

Achieving cutting-edge storage performance tends to require bypassing the filesystem anyways. Traditionally, that meant using SPDK. Nowadays, opening /dev/nvme* with O_DIRECT and operating on it with io_uring will get you most of the way there. In either case, the advice given in the article and by the OP is filesystem agnostic.

I'm wondering if it's really necessary to get at the block device directly.

I'm able to saturate a PCIe 3.0 x4 link doing direct IO to an NVMe drive with a single 1.7 GHz Power PC core without breaking a sweat. This is through ext4.

My accesses are sequential though. Maybe there's more of a penalty with random IO.

Re: What every programmer should know about SSDs

#148

Post by mikewarot »

qiqitori wrote: post #133
mikewarot wrote: post #114

If you leave un-partitioned space on the SSD, how the heck does the SSD know it is ok to erase it? Wouldn't it be safer to partition it as an extra drive letter, format it, and then leave that drive alone? That would allow the OS to trim all the "empty" blocks.

Not 100% sure what you are replying to, and not sure what you meant by "safer", but this may help: The actual physical address on the storage chip and the physical address from the operating system's perspective don't have much to do with another. For harddrives, "un-partitioned space" means that there is a physical "chunk of metal" that is unused. However, that's not the case for SSDs. SSDs dynamically remap "OS-phy…

>Not 100% sure what you are replying to, and not sure what you meant by "safer", but this may help:

I'm replying to the whole of comments on this article. The write amplification problem goes up as the number of "free" sectors/blocks goes down. Many solutions have been presented that don't allocate X% of the hard drive... but I'm not sure than any of them let the hard drive's SSD controller know they aren't allocated.

For that to happen, the OS has to have TRIM support, AND the block in question has to be on a volume that the OS is managing.

My worry is that if you have a blank partition, it's not being actively managed by anything, and thus isn't going to be TRIMed, and thus the SSD doesn't know the blocks are free for use.

Thus, leaving an unpartitioned area isn't going to help.

Re: What every programmer should know about SSDs

#149

Post by Shorel »

formerly_proven wrote: post #125
jedberg wrote: post #82

Earlier quoted context omitted.

Yes but you can configure the kernel to ignore that, and by default it does. For example, way back in the day, to get more life out of my laptop during college, I configured the kernel to only write to disk once an hour or when the buffer filled up. That effectively meant I was only writing to disk once per hour when I shut down to change classes. The modern linux kernel doesn't actually write to disk when fsync is c…

> The modern linux kernel doesn't actually write to disk when fsync is called. It buffers the writes in a cache. That's not true, you can tell in many ways but one of the easiest is because fsync is quite slow and noisy (on hard drives).

I would be a bit disappointed if the kernel implementation for HDD and SSD is exactly the same.

Re: What every programmer should know about SSDs

#150

Post by nh2 »

jedberg wrote: post #82
gruez wrote: post #81

Earlier quoted context omitted.

The kernel can't optimize that because sqlite is specifically requesting it to force a write.

Yes but you can configure the kernel to ignore that, and by default it does. For example, way back in the day, to get more life out of my laptop during college, I configured the kernel to only write to disk once an hour or when the buffer filled up. That effectively meant I was only writing to disk once per hour when I shut down to change classes. The modern linux kernel doesn't actually write to disk when fsync is c…

> you can configure the kernel to ignore that, and by default it does.

> The modern linux kernel doesn't actually write to disk when fsync is called.

This is false.

Almost all open source databases' durability guarantees are based upon fsync (including SQLite, Postgres, MySQL, and so on). fsync will result in the corresponding underlying storage flush commands. You configure Linux to ignore fsync, but this is is not the default, on any Linux distribution I'm aware of. It would not make any sense.

Post reply on HN