Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

61–70 of 115 posts

Re: LZ4 – Extremely fast compression

#61

Post by not2b »

YetAnotherNick wrote: post #57
unsigner wrote: post #38

Earlier quoted context omitted.

Zstd is very different - it includes an entropy coder. LZ4 only finds repeated matches, but then doesn't encode them very efficiently. To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly, while Zstd will compress it 8:1 converging to an encoding where a '1' bit is A, and a '0' bit is B.

> To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly I checked it. LZ4 is still reducing the size to half, no idea why half. So for 10 MB file it compresses to 5 MB. Edit: checked with highest compression and it compresses 1MB file to 185KB. So what the parent wrote is false.

Yes, if I take the 8 combinations aaa, aab, aba etc and assign each of them a 9 bit codeword I replace each 24 bit sequence with a 9 bit sequence. So arithmetic coders have no problem with cases like this.

Re: LZ4 – Extremely fast compression

#62

Post by andruby »

KitDuncan wrote: post #59
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

Probably not, but ZFS is actually smart enough to store the uncompressed version if compression doesn't save space. In other words: zfs will try lz4 compression on those video files, notice that it doesn't gain anything and store it uncompressed.

[0] https://klarasystems.com/articles/openzfs1-understanding-tra...

Re: LZ4 – Extremely fast compression

#63

Post by livvelupp »

YetAnotherNick wrote: post #57
unsigner wrote: post #38

Earlier quoted context omitted.

Zstd is very different - it includes an entropy coder. LZ4 only finds repeated matches, but then doesn't encode them very efficiently. To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly, while Zstd will compress it 8:1 converging to an encoding where a '1' bit is A, and a '0' bit is B.

> To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly I checked it. LZ4 is still reducing the size to half, no idea why half. So for 10 MB file it compresses to 5 MB. Edit: checked with highest compression and it compresses 1MB file to 185KB. So what the parent wrote is false.

YetAnother*ick

Re: LZ4 – Extremely fast compression

#64

Post by older »

st_goliath wrote: post #32

A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image. I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data…

F2FS supports compression with LZ4 since Linux 5.6.

Re: LZ4 – Extremely fast compression

#65

Post by garaetjjte »

gens wrote: post #55
garaetjjte wrote: post #50

Earlier quoted context omitted.

>and even BCn, can't be compressed much further S3TC is block compression, so if there is repeating data in images it will compress quite well.

I tried compressing BC7 (from AMD-s Compressonator) using lz4c and it wasn't much. Just re-ran it (with -hc, version 1.9.3(latest now)) and: "Compressed 5592544 bytes into 5413783 bytes ==> 96.80%". 7z with "7z a -mx=9 -t7z asd.7z albedo_PNG_BC7_1.KTX" does 5592544 bytes to 4844707 bytes (down to ~87%). Original file is 10227047 bytes (PNG, RGBA), i can't remember if the ktx has mipmaps. EDIT: Note that the image is…

Yes, it depends on content and won't do much for grainy surfaces. I don't have AAA game quality textures to compare, but I think for usual textured model it is still worthwhile. eg. this https://milek7.pl/.stuff/somelocotex.png weights 16MiB uncompressed, 5.1MiB as PNG, 5.3MiB as DXT5, and 2.1MiB as DXT5-in-LZ4. (mipmaps included in DXT5)

>PS I think that BC could be massaged at compression to be better compressible, and i think i read something about that. Don't remember.

There's Oodle BC7Prep from RAD Game Tools: http://www.radgametools.com/oodletexture.htm

EDIT: RAD page states that "BC7 blocks that are often very difficult to compress", so that might be also a factor why my DXT5 test compressed much better than your BC7.

EDIT2: Yeah, with BC7 LZ4 only compresses it down to 4.6MiB.

Re: LZ4 – Extremely fast compression

#66

Post by jakozaur »

KitDuncan wrote: post #59
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

LZ4 is not going to help, if you already have compressed data.

For image, video and audio there are more efficient compressions taking advantage of those formats.

Re: LZ4 – Extremely fast compression

#67

Post by Mashimo »

KitDuncan wrote: post #59
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

I compression not on by default nowdays? Anyhow, I would not run ZFS with compression disabled completely. There are edgecases where you want it. The meta data? I can't remember the details. At least active the compression that just compresses zeros.

Re: LZ4 – Extremely fast compression

#68

Post by tyingq »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

"Even storing items in-memory compressed sometimes is profitable"

LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

Re: LZ4 – Extremely fast compression

#70

Post by TwoBit »

maeln wrote: post #17

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

How does it compare to Kraken? It was fast enough that Sony recently built Kraken into the PS5 hardware.
Post reply on HN