Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduce ZSTD compression to ZFS

github.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Introduce ZSTD compression to ZFS

#21

Post by tester756 »

jerryX wrote: post #14
rakoo wrote: post #12

Earlier quoted context omitted.

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…

Zstd uses this new ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems

Guy who stands behind this has insane CV

2015- Jagiellonian University, Institute of Computer Science, assistant professor,

2013-2014 Purdue University, NSF Center for Science of Information, Postdoctoral researcher (webpage),

2006-2012 Jagiellonian University, Cracow, PhD in Theoretical Physics (thesis)

2004-2010 Jagiellonian University, Cracow, PhD in Theoretical Computer Science (thesis)

2001-2006 Jagiellonian University, Cracow, MSc in Theoretical Physics (thesis)

2000-2005 Jagiellonian University, Cracow, MSc in Theoretical Mathematics (thesis)

1999-2004 Jagiellonian University, Cracow, MSc in Computer Science (thesis)

Re: Introduce ZSTD compression to ZFS

#22

Post by snvzz »

db48x wrote: post #10
chungy wrote: post #9

Earlier quoted context omitted.

I think it's likely that LZ4 remains the default. There are almost no downsides to having LZ4 enabled. zstd is more likely to represent an obsolescence of gzip. It surpasses gzip pretty much always.

Just going by those graphs, I could double my compression ratio by going from lz4 to zstd-1 without going below the speeds the drives in my pool can manage. The usual caveats apply, but it seems to me that this is a pretty good upgrade for the usual case where you're using hard drives instead of fast ssds in a pool.

This sounds good for fileservers. But generally speaking, computers usually do more than i/o.

Re: Introduce ZSTD compression to ZFS

#23

Post by cmurf »

tpetry wrote: post #13

An option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.

Fast compression levels of a given algorithm means lower compression ratio.

I don't know if ZFS supports variable compression levels (maybe per dataset), but Btrfs ZSTD support uses a mount option, e.g. mount -o compress=zstd:[1-15]

Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when doing updates. Writes use the compression algorithm and level set at mount time; and it's possible to change it while remaining mounted, using -o remount.

Re: Introduce ZSTD compression to ZFS

#24

Post by rakoo »

jfkebwjsbx wrote: post #19
rakoo wrote: post #12

Earlier quoted context omitted.

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

Depends what "read many times" means.

- If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages (https://www.archlinux.org/news/now-using-zstandard-instead-o...)

- If you mean that the same archive will be read multiple times by the same machine, I don't really know what kind of scenario that is; I'd deflate the archive into its initial representation once and then let processes access that folder directly. Note that zstd claims that it isn't that much slower in decompression than competitors, even if you always use compressed archives the difference will be minimal

zstd was built more or less to "replace" all formats that favor compression over speed. From their benchmarks (which means what it means) whatever the compression/speed ratio you want, zstd is going to be better than all of them, with a hard exception on extremely fast speed that is still the kingdom of lz4.

Re: Introduce ZSTD compression to ZFS

#25

Post by sigstoat »

cmurf wrote: post #23
tpetry wrote: post #13

An option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.

Fast compression levels of a given algorithm means lower compression ratio. I don't know if ZFS supports variable compression levels (maybe per dataset), but Btrfs ZSTD support uses a mount option, e.g. mount -o compress=zstd:[1-15] Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when do…

> I don't know if ZFS supports variable compression levels (maybe per dataset)

> Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when doing updates.

yup. works the same in ZFS. you can change the compression setting any time you like, for future writes.

Re: Introduce ZSTD compression to ZFS

#26

Post by shockinglytrue »

jerryX wrote: post #14
rakoo wrote: post #12

Earlier quoted context omitted.

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…

Zstd uses this new ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems

IIRC it doesn't use ANS for every compression level, can't remember much more than that, though

Re: Introduce ZSTD compression to ZFS

#27

Post by microcolonel »

d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

> Why isn't the implementation of the algorithm as simple as linking in the related library

Because filesystems need to store compression metadata and compressor settings differently than individual archive streams do. Because of the way ZFS stores configuration like this, in the previous version of these patches, they had to choose a subset of the available compression levels when adding zstd support.

Different compressors and decompressors also have different state sizes for different settings. Allocating, reusing, and discarding buffers for compression/decompression state in a sensible way inside an operating system kernel is not trivial.

Re: Introduce ZSTD compression to ZFS

#28

Post by microcolonel »

rakoo wrote: post #24
jfkebwjsbx wrote: post #19

Earlier quoted context omitted.

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages ( https://www.archlinux.org/news/now-using-z…

It also depends on your workload and the speed of your disks.

If your disks are faster than your decompression algorithm when that algorithm is running alongside the rest of your workload (generally not the case) then it can make sense to use the faster decompressor (lz4). In my understanding of the tradeoffs of zstd though, having used it recently in an application, chances are you have a free hardware thread that can saturate your disk without affecting your compute workload.

Re: Introduce ZSTD compression to ZFS

#29

Post by jfkebwjsbx »

rakoo wrote: post #24
jfkebwjsbx wrote: post #19

Earlier quoted context omitted.

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages ( https://www.archlinux.org/news/now-using-z…

By read many times I mean a file that is opened many times. For instance, the kernel and libraries every time you boot.

From my understanding (but please correct me if I am wrong), LZ4 will decompress them significantly faster than ZSTD even if the latter compresses more.

In other words, the decompression speed is measured on the decompressed data, right?

Re: Introduce ZSTD compression to ZFS

#30

Post by jfkebwjsbx »

microcolonel wrote: post #28
rakoo wrote: post #24

Earlier quoted context omitted.

Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages ( https://www.archlinux.org/news/now-using-z…

It also depends on your workload and the speed of your disks. If your disks are faster than your decompression algorithm when that algorithm is running alongside the rest of your workload (generally not the case) then it can make sense to use the faster decompressor (lz4). In my understanding of the tradeoffs of zstd though, having used it recently in an application, chances are you have a free hardware thread that c…

Considering that nowadays many people have an SSD, for boot files that would mean LZ4 is best.

But perhaps for your data files that you don't open often, ZSTD is best because you save space on the SSD.

Post reply on HN