LZ4 – Extremely fast compression • Hacker News Board

Re: LZ4 – Extremely fast compression

#71

Post by pmarreck » Wed, Jan 27, 2021, 4:20 PM UTC

tyingq wrote: post #68

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

"Even storing items in-memory compressed sometimes is profitable" LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

TIL about zRAM, emailed myself this config to try at work

https://www.techrepublic.com/article/how-to-enable-the-zram-...

Re: LZ4 – Extremely fast compression

#72

Post by pmarreck » Wed, Jan 27, 2021, 4:25 PM UTC

StreamBright wrote: post #7

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

> Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data Yes, however there is usually no facility to train your compression algo with most tools using ZSTD.

There should be a way to pool standard dictionaries somewhere, such as a "standard english text corpus data" dictionary, that you can then download on demand for encoding, say, BLOB text fields in a database with little to no overhead.

The way this would probably work without this facility though, say, in a database, is that the dictionary is maintained internally and constructed on the fly from the field data and not exposed to users. Although, I don't know if you'd have to keep every version of the dictionary in order to successfully decompress old data? If so then perhaps this is a niche feature

Re: LZ4 – Extremely fast compression

#73

Post by pmarreck » Wed, Jan 27, 2021, 4:28 PM UTC

PSeitz wrote: post #33

I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex

NICE! Well done!

Re: LZ4 – Extremely fast compression

#74

Post by 0-_-0 » Wed, Jan 27, 2021, 4:40 PM UTC

rav wrote: post #54

Jonnax wrote: post #22

Earlier quoted context omitted.

The frustrating thing about 7zip is that whilst it's opensource, a single author just does code drops. So there's a bunch of forks with useful features that'll never be adopted because there's no collaboration. At least that's what I could tell when I looked into it

It sounds like it needs a fork to combine all the forks, similar to what Neovim did as a fork of Vim.

Would that be like... a reverse fork? A defork? A korf?

Re: LZ4 – Extremely fast compression

#75

Post by Artlav » Wed, Jan 27, 2021, 4:46 PM UTC

bombcar wrote: post #6

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

Predictability, no? Sometimes you want to know how large your data really is if it was to get onto an uncompressed filesystem.

Re: LZ4 – Extremely fast compression

#76

Post by thechao » Wed, Jan 27, 2021, 4:57 PM UTC

smueller1234 wrote: post #39

julian37 wrote: post #5

Earlier quoted context omitted.

Yep, Zstd is the spiritual successor to LZ4 and written by the same person (Yann Collet) after they got hired by Facebook.

Actually, I seem to recall that he was working on it before getting hired by Facebook (unless there was a massive delay in the hiring to become known). I was following his excellent blog posts on the matter at the time.

Yann wrote LZ4 and Zstd well before joining FB. I have to applaud FB for supporting Yann's work, though.

Re: LZ4 – Extremely fast compression

#77

Post by 1996 » Wed, Jan 27, 2021, 5:04 PM UTC

st_goliath wrote: post #32

A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image. I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data…

> lz4 (...) probably quite suitable for life filesystems and network protocols

Actually, no. lz4 is less suitable than zstd for filesystems.

BTW, lz4 is present in many mozilla tools like thunderbird: it's represented by its bastard child lz4json, which is diverging by just the headers don't work with regular lz4 tools

> achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my

Your observation is correct: zstd is now standard and the default on openzfs 2.0, replacing lz4.

The 19 compression variants offer more flexibility than just lz4- another strength is the decode time is not a function of the compression factor, which is something good on coldish storage that's rarely updated.

Re: LZ4 – Extremely fast compression

#78

Post by nyanpasu64 » Wed, Jan 27, 2021, 5:05 PM UTC

Jonnax wrote: post #22

AnthonBerg wrote: post #2

Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/ (Linking to this more for the overview than the Windows tool in itself.)

The frustrating thing about 7zip is that whilst it's opensource, a single author just does code drops. So there's a bunch of forks with useful features that'll never be adopted because there's no collaboration. At least that's what I could tell when I looked into it

I liked http://www.e7z.org/ for its UI improvements, but it was last updated in 2016 and misses out on several security bug patches.

Re: LZ4 – Extremely fast compression

#79

Post by tpetry » Wed, Jan 27, 2021, 5:17 PM UTC

stavros wrote: post #56

tpetry wrote: post #13

Earlier quoted context omitted.

Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...

Do you mean LZ4? I fear your comment is a bit misleading as-is.

Ups, yeah it should be lz4, but i can‘t change it anymore.

Re: LZ4 – Extremely fast compression

#80

Post by tmd83 » Wed, Jan 27, 2021, 5:18 PM UTC

smueller1234 wrote: post #39

julian37 wrote: post #5

Earlier quoted context omitted.

Yep, Zstd is the spiritual successor to LZ4 and written by the same person (Yann Collet) after they got hired by Facebook.

Actually, I seem to recall that he was working on it before getting hired by Facebook (unless there was a massive delay in the hiring to become known). I was following his excellent blog posts on the matter at the time.

Yes it was a fully working things before facebook. There has been a lot of improvement in both the core and cli. But the core innovations of zstd was well established before facebook. I was probably following his blogs (even though I wasn't a compression expert) for months before I saw the post about his joining facebook.