Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduce ZSTD compression to ZFS

github.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Introduce ZSTD compression to ZFS

#31

Post by vkaku »

d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

It's usually the same as linking, except there are tunables in every library, and the hard part is figuring out which one is appropriate for the job. Maybe, this call is to ask people how this library works out in the real world, on the current datasets that people have. Besides, they are talking of testing implementation specific limitations, and may they want to test for that.

Re: Introduce ZSTD compression to ZFS

#32

Post by foepys »

the8472 wrote: post #20
jfkebwjsbx wrote: post #19

Earlier quoted context omitted.

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

xz would be an example of the latter kind.

Does xz actually outperform ZSTD anywhere? IIRC even at high compression ratios ZSTD is a lot faster than xz.

Edit: Arch Linux switched from xz to ZSTD for their package manager and somebody compared both: https://sysdfree.wordpress.com/2020/01/04/293/

The Arch Linux developers state that they expect an 0.8% you increase in package size but an 1300% speedup in decompression. Not too shabby.

I'm running Arch on my personal system and it's really noticable, especially when I create by own packages, compression doesn't take longer than compiling anymore.

Re: Introduce ZSTD compression to ZFS

#33

Post by [deleted] »

rcthompson wrote: post #15
tpetry wrote: post #13

An option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.

I think that would require the same architectural changes needed for offline deduplication (i.e. probably not going to happen any time soon, unless I've missed some recent developements).

[deleted]

Re: Introduce ZSTD compression to ZFS

#34

Post by microcolonel »

jfkebwjsbx wrote: post #30
microcolonel wrote: post #28

Earlier quoted context omitted.

It also depends on your workload and the speed of your disks. If your disks are faster than your decompression algorithm when that algorithm is running alongside the rest of your workload (generally not the case) then it can make sense to use the faster decompressor (lz4). In my understanding of the tradeoffs of zstd though, having used it recently in an application, chances are you have a free hardware thread that c…

Considering that nowadays many people have an SSD, for boot files that would mean LZ4 is best. But perhaps for your data files that you don't open often, ZSTD is best because you save space on the SSD.

> Considering that nowadays many people have an SSD, for boot files that would mean LZ4 is best.

That depends on how concurrent boot is, and how fast your CPU and memory are. It may be true on a Celeron, but maybe not on a ThreadRipper.

Furthermore, if you look at the performance testing, the sequential read performance was almost always better with zstd than with lz4, in ZFS; and the zstd-fast mode was about as fast as the lz4 mode in sequential writes. This may be a matter of their specific integration of lz4, but nonetheless it pays to look at the actual numbers before drawing conclusions.

Re: Introduce ZSTD compression to ZFS

#35

Post by usefulcat »

jfkebwjsbx wrote: post #19
rakoo wrote: post #12

Earlier quoted context omitted.

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

It depends on whether read rates are more likely to be limited by CPU or by disk speed. If the former, you'll probably want something like LZ4 to minimize CPU usage. If the latter, it may be better to use a more CPU intensive algorithm which provides better compression, so that there is less compressed data to read from the disk.

Re: Introduce ZSTD compression to ZFS

#36

Post by usefulcat »

foepys wrote: post #32
the8472 wrote: post #20

Earlier quoted context omitted.

xz would be an example of the latter kind.

Does xz actually outperform ZSTD anywhere? IIRC even at high compression ratios ZSTD is a lot faster than xz. Edit: Arch Linux switched from xz to ZSTD for their package manager and somebody compared both: https://sysdfree.wordpress.com/2020/01/04/293/ The Arch Linux developers state that they expect an 0.8% you increase in package size but an 1300% speedup in decompression. Not too shabby. I'm running Arch on my per…

There definitely exist certain kinds of data which xz can compress better or as well but faster than zstd, even taking into account the more extreme compression options offered by zstd. I know because we have many terabytes of such data, and I've done thorough comparisons between xz and zstd at all different compression levels.

However in all cases zstd is much faster for decompression. It just happens that getting the best compression ratio in a not insane amount of time is still the better tradeoff for us.

Re: Introduce ZSTD compression to ZFS

#37

Post by rakoo »

microcolonel wrote: post #28
rakoo wrote: post #24

Earlier quoted context omitted.

Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages ( https://www.archlinux.org/news/now-using-z…

It also depends on your workload and the speed of your disks. If your disks are faster than your decompression algorithm when that algorithm is running alongside the rest of your workload (generally not the case) then it can make sense to use the faster decompressor (lz4). In my understanding of the tradeoffs of zstd though, having used it recently in an application, chances are you have a free hardware thread that c…

Indeed in these cases lz4 makes sense. I'm thinking of what lz4 was created to "replace" namely snappy in SSTables: the pattern is mostly write once, read at most once or twice, and both of those are sequential to benefit of non-SSDs speeds. In these situations fast (de)compression schemes are definitely a gain.

Re: Introduce ZSTD compression to ZFS

#38

Post by namibj »

jfkebwjsbx wrote: post #29
rakoo wrote: post #24

Earlier quoted context omitted.

Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages ( https://www.archlinux.org/news/now-using-z…

By read many times I mean a file that is opened many times. For instance, the kernel and libraries every time you boot. From my understanding (but please correct me if I am wrong), LZ4 will decompress them significantly faster than ZSTD even if the latter compresses more. In other words, the decompression speed is measured on the decompressed data, right?

Yes, speeds are measured relative to the uncompressed data. ZSTD outputs around 600-2000MiB/s/core (in all the higher levels), depending on how fast your processor is.

Re: Introduce ZSTD compression to ZFS

#39

Post by namibj »

cmurf wrote: post #23
tpetry wrote: post #13

An option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.

Fast compression levels of a given algorithm means lower compression ratio. I don't know if ZFS supports variable compression levels (maybe per dataset), but Btrfs ZSTD support uses a mount option, e.g. mount -o compress=zstd:[1-15] Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when do…

Btrfs is soon getting support for specifying the level on a per-file basis: https://github.com/kdave/btrfs-progs/issues/184

Re: Introduce ZSTD compression to ZFS

#40

Post by glandium »

tobias3 wrote: post #7
d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

In this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse. W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD…

Zstd is dual licensed BSD and GPL. It is kind of weird the kernel limits it to GPL modules.
Post reply on HN