Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduce ZSTD compression to ZFS

github.com

Post reply on HN
Print view View original

1–10 of 41 posts

Re: Introduce ZSTD compression to ZFS

#2

Post by d33 »

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexity from the API/implementation perspective given that a library is already there?

Re: Introduce ZSTD compression to ZFS

#3

Post by georgyo »

d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

For one thing, kernel models don't link against libraries. So a new implementation is almost always written.

Re: Introduce ZSTD compression to ZFS

#4

Post by Someone »

georgyo wrote: post #3
d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

For one thing, kernel models don't link against libraries. So a new implementation is almost always written.

This incorporates the zstd source code ‘as is’. FTA:

Frequently Asked Questions

“Q: Why is it so many lines of code, I can't review all of that...

A: Most of this code is the ZSTD library, which has not been altered.”

One reason this needs testing because this is a file system. If it breaks, it can lose you much more data than the file being worked on.

There also may be serious performance degradation on hardware or configurations the developers didn’t look at.

There also is some new code added to call the zstd library.

Re: Introduce ZSTD compression to ZFS

#7

Post by tobias3 »

d33 wrote: post #2

There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…

In this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse.

W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it directly. Maybe all future compression algorithms can provide a ZSTD compatible API, so we have to do less work in the future?

Re: Introduce ZSTD compression to ZFS

#9

Post by chungy »

foepys wrote: post #8

lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.

I think it's likely that LZ4 remains the default. There are almost no downsides to having LZ4 enabled.

zstd is more likely to represent an obsolescence of gzip. It surpasses gzip pretty much always.

Re: Introduce ZSTD compression to ZFS

#10

Post by db48x »

chungy wrote: post #9
foepys wrote: post #8

lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.

I think it's likely that LZ4 remains the default. There are almost no downsides to having LZ4 enabled. zstd is more likely to represent an obsolescence of gzip. It surpasses gzip pretty much always.

Just going by those graphs, I could double my compression ratio by going from lz4 to zstd-1 without going below the speeds the drives in my pool can manage. The usual caveats apply, but it seems to me that this is a pretty good upgrade for the usual case where you're using hard drives instead of fast ssds in a pool.
Post reply on HN