How does it work?
I found this blog post describing some of the mechanics, but I am still interested in reading how this may be implemented branchlessly.
LZ4 – Extremely fast compression
31–40 of 115 posts
Re: LZ4 – Extremely fast compression
#32A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image.
I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least).
[1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...
[2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...
[3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...
Re: LZ4 – Extremely fast compression
#33I ported the block format to Rust matching the C implementation in performance and ratio.
Re: LZ4 – Extremely fast compression
#34The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.
As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s
Re: LZ4 – Extremely fast compression
#35Earlier quoted context omitted.
I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?
Yes. Much faster. Especially for HDDs. But at a cost of a lot of RAM. Also lz4 compression can speed up your HDDs up to 10x (!) to read and 3x to write. [1, see "To compress, or not to compress, that is the question" section.] But it's going to have a considerably higher CPU usage as well. [1]: https://calomel.org/zfs_raid_speed_capacity.html
>But it's going to have a considerably higher CPU usage as well.
I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15.
Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )
Re: LZ4 – Extremely fast compression
#36The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.
You misread that. It says that decompression is significantly faster than LZO. In fact, decompression speeds of up to 5 GB/s is one of the reasons LZ4 is so popular.
Re: LZ4 – Extremely fast compression
#37How does it work?
Ring buffer and huffman codes?
Re: LZ4 – Extremely fast compression
#38another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.
Zstd is very different - it includes an entropy coder.
LZ4 only finds repeated matches, but then doesn't encode them very efficiently.
To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly, while Zstd will compress it 8:1 converging to an encoding where a '1' bit is A, and a '0' bit is B.
Re: LZ4 – Extremely fast compression
#39another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.
Yep, Zstd is the spiritual successor to LZ4 and written by the same person (Yann Collet) after they got hired by Facebook.
Actually, I seem to recall that he was working on it before getting hired by Facebook (unless there was a massive delay in the hiring to become known). I was following his excellent blog posts on the matter at the time.
Re: LZ4 – Extremely fast compression
#40Not sure how LZ4 compares but what convinced me to use brotli over some other compression libraries was how trivialy easy it was to integrate in C++ as well as in javascript. No effort at all. I wish more lib providers would make their things so approachable.