11–20 of 41 posts
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like file transfers for example
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…
An option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.
There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…
In this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse. W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD…
Could it? That doesn't sound very portable, and ZFS works on FreeBSD too.
There's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexit…
See BSDCan 2018 "Implementing ZSTD in OpenZFS on FreeBSD" by Allan Jude: * https://www.bsdcan.org/2018/schedule/events/947.en.html * https://twitter.com/allanjude
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…
From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read.
And I'd assume there are other algorithms that focus only on size, trading speed for it.
Earlier quoted context omitted.
FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like f…
Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.