What every programmer should know about SSDs • Hacker News Board

Re: What every programmer should know about SSDs

#161

Post by hyc_symas » Tue, Jun 22, 2021, 6:16 PM UTC

ww520 wrote: post #136

fulafel wrote: post #117

Earlier quoted context omitted.

This is quite common in traditional DBs too. Eg PostgreSQL has its write-ahead log. Both LMDB and PostgreSQL then occasionally need to do do some kind of compaction, checkpoint or garbage collection, whatever it's called in various systems, the write-only log is reset and any live data in it improted into the main db data.

I only have a cursory knowledge on LMDB (listening to a podcast while biking). Anyway, LMDB has no transaction log nor write ahead log. There's no overwrite during update. Data page update is copy-on-write and b+tree index update is append only. The update on the b+tree pages is performed from the bottom of the tree to the root, linking newly appended pages to higher level pages. The transaction is committed when the…

No. LMDB is copy-on-write, with double buffering/shadow pages for the root page updates. No searching for the last valid root page.

Looks like you have the other details right.

Re: What every programmer should know about SSDs

#162

Post by ssvss » Wed, Jun 23, 2021, 9:29 AM UTC

pkaye wrote: post #89

jng wrote: post #67

Earlier quoted context omitted.

Hey that's very interesting! How much of the FTL logic is done with regular MCU code vs custom hardware? Is there any open source SSD firmware out there that one could look at to start experimenting in this field, or at least something pointing in that direction, be it open or affordable software, firmware, FPGA gateway or even IC IP? I believe there is value in integrating that part of the stack with the higher leve…

Typically the Host and NAND interface have custom hardware. When the host issues a command, the hardware might validate it and queue up data to a buffer. On the NAND interface there might be a similar queue for NAND commands. You might have multiple queues for different priorities of operation. The error correct will also be in hardware. When you issues NAND reads and writes, the ECC will be checked or encoded. The r…

Do you know if Apple's M1 also does something similar to what is done by Fusion IO. I read something about it in Twitter, but didn't think to follow up with the twitter poster at that time.

Re: What every programmer should know about SSDs

#163

Post by raffraffraff » Wed, Jun 23, 2021, 11:51 AM UTC

barrkel wrote: post #143

raffraffraff wrote: post #128

Earlier quoted context omitted.

Isn't this extremely dangerous? Disk write caches aren't used most of the time, except on battery backed HBAs. And databases are typically configured to use O_DIRECT for a reason: COMMITs are supposed to be durable. We had this fight at a previous company when an engineer based database server hardware recommendation on a dangerously misconfigured database server, and did not consider the effect of caches. As soon as…

Parent is talking about temporary tables. Those are normally only live for the duration of a transaction (well, session, but in practice if you're using temporary tables across multiple transactions you have a logical application-level transaction which needs to be able to handle failure part-way through). After your transaction the writes to non-temporary tables should be persistent. Postgres temp tables on ramdisk…

Gotcha, somehow missed that. Yeah, tmp tables on disks are painful and I've made the same optimization on MySQL whenever it wasn't possible to eliminate the need to tmp tables by refactoring SQL.