LZ4 – Extremely fast compression • Hacker News Board

LZ4 – Extremely fast compression

#1

Post by marcobambini » Tue, Jan 26, 2021, 12:56 PM UTC

LZ4 – Extremely fast compression
lz4.github.io

Re: LZ4 – Extremely fast compression

#2

Post by AnthonBerg » Wed, Jan 27, 2021, 9:29 AM UTC

Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/

(Linking to this more for the overview than the Windows tool in itself.)

Re: LZ4 – Extremely fast compression

#3

Post by roncohen » Wed, Jan 27, 2021, 9:32 AM UTC

another contender is zstd: https://github.com/facebook/zstd. It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

Re: LZ4 – Extremely fast compression

#4

Post by moonchild » Wed, Jan 27, 2021, 9:36 AM UTC

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed

Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4.

They both have their place.

0. https://facebook.github.io/zstd/

Re: LZ4 – Extremely fast compression

#5

Post by julian37 » Wed, Jan 27, 2021, 9:36 AM UTC

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

Yep, Zstd is the spiritual successor to LZ4 and written by the same person (Yann Collet) after they got hired by Facebook.

Re: LZ4 – Extremely fast compression

#6

Post by bombcar » Wed, Jan 27, 2021, 9:50 AM UTC

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

Re: LZ4 – Extremely fast compression

#7

Post by StreamBright » Wed, Jan 27, 2021, 9:52 AM UTC

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

> Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data

Yes, however there is usually no facility to train your compression algo with most tools using ZSTD.

Re: LZ4 – Extremely fast compression

#8

Post by viktorcode » Wed, Jan 27, 2021, 9:54 AM UTC

Weird that the page doesn't list macOS or other Apple's OS in the list of operating systems with LZ4 support.

Re: LZ4 – Extremely fast compression

#9

Post by baybal2 » Wed, Jan 27, 2021, 10:06 AM UTC

moonchild wrote: post #4

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/

LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well.

zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.

Re: LZ4 – Extremely fast compression

#10

Post by _vvhw » Wed, Jan 27, 2021, 10:12 AM UTC

baybal2 wrote: post #9

moonchild wrote: post #4

Earlier quoted context omitted.

> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/

LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well. zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.

I also appreciate LZ4's simplicity and tiny code footprint.

zstd is brilliant as well, but in terms of code base it's a whole other beast.