Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every programmer should know about SSDs

databasearchitects.blogspot.com

Post reply on HN
Print view View original

21–30 of 163 posts

Re: What every programmer should know about SSDs

#21

Post by forrestthewoods »

andrewmcwatters wrote: post #14

My opinion is probably... not technically correct... until you have to deal with drive reliability and write guarantees, but I don't think programmers actually have to know anything about SSDs in the same way that developers had to know particular things about HDDs. This is out of pure speculation, but there had to be a period of time during the mass transition to SSDs that engineers said, OK, how do we get the hardw…

Games used to spend a lot time optimizing CD/DVD layout. Because reading from that is REALLY slow. Optimize mostly meant keep data contiguous. But sometimes it meant duplicate data to avoid seeks.

The canonical case is minimize time to load a level. Keep that level’s assets contiguous. And maybe duplicate data that is shared across levels. It’s a trade off between disc space and load time.

I’m not familiar with major tricks for improving after a disc is installed to drive. (PS4 games always streamed data from HDD, not disc.)

Even consoles use different HDD manufacturers. So it’d be pretty difficult to safely optimize for that. I’m sure a few games do. But it’s rare enough I’ve never heard of it.

Re: What every programmer should know about SSDs

#22

Post by FpUser »

It is really puzzling why "every programmer" should burden their already overloaded brains with this. If they're reading/writing some config/data files this knowledge would not help one bit. If they're using database then it falls to the database vendor's to optimize for this scenario.

So I think that unless this "every programmer" is a database storage engine developer (not too many of them I guess) their only concern would be mostly - how close my SSD to that magical point where it has to be cloned and replaced before shit hits the fan.

Re: What every programmer should know about SSDs

#23

Post by andrewmcwatters »

forrestthewoods wrote: post #21
andrewmcwatters wrote: post #14

My opinion is probably... not technically correct... until you have to deal with drive reliability and write guarantees, but I don't think programmers actually have to know anything about SSDs in the same way that developers had to know particular things about HDDs. This is out of pure speculation, but there had to be a period of time during the mass transition to SSDs that engineers said, OK, how do we get the hardw…

Games used to spend a lot time optimizing CD/DVD layout. Because reading from that is REALLY slow. Optimize mostly meant keep data contiguous. But sometimes it meant duplicate data to avoid seeks. The canonical case is minimize time to load a level. Keep that level’s assets contiguous. And maybe duplicate data that is shared across levels. It’s a trade off between disc space and load time. I’m not familiar with major…

This reminds me of Valve’s GCF (grid cache file, officially, or game cache file, commonly). The benefits must have purely occurred on consoles for the reasons you outlined, because cracked Valve games that had GCF files extracted ran faster than the official retail releases on PCs!

Re: What every programmer should know about SSDs

#24

Post by alpaca128 »

forrestthewoods wrote: post #21
andrewmcwatters wrote: post #14

My opinion is probably... not technically correct... until you have to deal with drive reliability and write guarantees, but I don't think programmers actually have to know anything about SSDs in the same way that developers had to know particular things about HDDs. This is out of pure speculation, but there had to be a period of time during the mass transition to SSDs that engineers said, OK, how do we get the hardw…

Games used to spend a lot time optimizing CD/DVD layout. Because reading from that is REALLY slow. Optimize mostly meant keep data contiguous. But sometimes it meant duplicate data to avoid seeks. The canonical case is minimize time to load a level. Keep that level’s assets contiguous. And maybe duplicate data that is shared across levels. It’s a trade off between disc space and load time. I’m not familiar with major…

Consoles also do this with HDDs. That's been one of the talking points around the PS5 from the beginning, with Sony saying that games would get more storage space efficient because they don't need redundancy for faster loading anymore.

Re: What every programmer should know about SSDs

#25

Post by pkaye »

dataflow wrote: post #5

What's the flash translation layer made of? Is the flash technology used for that more durable than the rest of the SSD itself? (like say MLC vs. QLC?)

The FTL is like a virtual memory manager. It is firmware/hardware to manage things like the logical to physical mapping table, garbage collection, error correction, bad block management. Yes there will be a lot of FTL data structures stored on the flash. It can be made durable by redundant copies, writing in SLC mode or having recovery algorithms. I used to develop SSD firmware in the past if you have further questions.

Re: What every programmer should know about SSDs

#26

Post by bob1029 »

Things I have learned about SSDs:

If you want to go fast & save NAND lifetime, use append-only log structures.

If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your block size is still a constraint when working in hardware. If you batch in software, you can hypothetically write multiple logical business transactions per block I/O. When you physical block size is 4k and your logical transactions are averaging 512b of data, you would be leaving a lot of throughput on the table.

Going down 1 level of abstraction seems important if you want to extract the most performance from an SSD. Unsurprisingly, the above ideas also make ordinary magnetic disk drives more performant & potentially last longer.

Re: What every programmer should know about SSDs

#27

Post by maccard »

alpaca128 wrote: post #24
forrestthewoods wrote: post #21

Earlier quoted context omitted.

Games used to spend a lot time optimizing CD/DVD layout. Because reading from that is REALLY slow. Optimize mostly meant keep data contiguous. But sometimes it meant duplicate data to avoid seeks. The canonical case is minimize time to load a level. Keep that level’s assets contiguous. And maybe duplicate data that is shared across levels. It’s a trade off between disc space and load time. I’m not familiar with major…

Consoles also do this with HDDs. That's been one of the talking points around the PS5 from the beginning, with Sony saying that games would get more storage space efficient because they don't need redundancy for faster loading anymore.

This is very very true. The PS5 does hardware decompression, so games by default are now going to be compressed. For a real world reference of how big a difference that makes, see fortnite turning on compression [0] (disclaimer: I worked for epic on fortnite at the time)

[0] https://www.ign.com/articles/fortnites-latest-patch-makes-it...

Re: What every programmer should know about SSDs

#28

Post by DrNuke »

A number of high-level techniques help rationalize data management and transfer, but the mileage of practical implementations may vary a lot. Generally speaking, only a small number of applications really need to take care and add a further layer of abstraction, that because the best practices already codified into any widespread language do an acceptable job already.

Re: What every programmer should know about SSDs

#29

Post by danbst »

jedberg wrote: post #16

This page tells me a lot about SSDs, but it doesn't tell me why I need to know these things. It doesn't really give me any indication about how I should change my behavior if I know that I'll be running on SSD vs spinning disk. I've always been told, "just treat SSDs like slow, permanent memory".

yeah, article should talk about periodic TRIMming, though this is more an admin advice

Re: What every programmer should know about SSDs

#30

Post by mercora »

teddyh wrote: post #19

What everyone should know is that flash drives can lose their data when left unpowered for as little as three months.

if that is true disks should come with a very visible note stating this... seriously, 3 months would be nothing. i doubt it is true because 3 months is a time frame which should be surpassed quite often making this more known.
Post reply on HN