Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

91–100 of 115 posts

Re: LZ4 – Extremely fast compression

#91

Post by zamalek »

tyingq wrote: post #68
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

"Even storing items in-memory compressed sometimes is profitable" LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

Microsoft found that compressed pages were always faster, because added by de-/compression was less than the latency to disk (given a sufficiently fast compression algorithm). As a bonus, it's also faster to read and write compressed pages to disk (if that absolutely has to happen). Zswap is therefore enabled by default on Windows.

I configure my own kernel on Arch and Zswap is enabled by default there, too.

Re: LZ4 – Extremely fast compression

#92

Post by felixhandte »

pmarreck wrote: post #72
StreamBright wrote: post #7

Earlier quoted context omitted.

> Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data Yes, however there is usually no facility to train your compression algo with most tools using ZSTD.

There should be a way to pool standard dictionaries somewhere, such as a "standard english text corpus data" dictionary, that you can then download on demand for encoding, say, BLOB text fields in a database with little to no overhead. The way this would probably work without this facility though, say, in a database, is that the dictionary is maintained internally and constructed on the fly from the field data and no…

W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-contradiction.

And yes, totally, I know at least RocksDB supports exactly that behavior [0].

[0] https://github.com/facebook/rocksdb/blob/12f11373554af219c51...

Re: LZ4 – Extremely fast compression

#93

Post by beefok »

I've been hunting for a good decompressor to use in a low ram microcontroller, for instance, an ARM Cortex M0. I've read an article [1] on LZ4 decompression on the Cortex, but I couldn't understand what kind of memory requirements are needed.

I've yet to really understand what kind of footprint LZ4 uses, and if it's dependent on dictionary size used to compress. What if I have, say, 4KB that I could use to store in-place decompression. Is that related to the compression ratio?

[1] https://community.arm.com/developer/ip-products/processors/b...

Re: LZ4 – Extremely fast compression

#94

Post by st_goliath »

yjftsjthsd-h wrote: post #90
1996 wrote: post #77

Earlier quoted context omitted.

> lz4 (...) probably quite suitable for life filesystems and network protocols Actually, no. lz4 is less suitable than zstd for filesystems. BTW, lz4 is present in many mozilla tools like thunderbird: it's represented by its bastard child lz4json, which is diverging by just the headers don't work with regular lz4 tools > achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my Yo…

> Actually, no. lz4 is less suitable than zstd for filesystems. Why's that? What benefit would I get from switching? Is it workload-dependent? EDIT: To be clear, I'm not disagreeing; if zstd will work better, I want to know about it so that I can switch my pools to use it.

>> Actually, no. lz4 is less suitable than zstd for filesystems.

>

>Why's that? What benefit would I get from switching? Is it workload-dependent?

Presumably because Zstd has much better compression, while still being quite fast.

I don't see however how that invalidates any of my observations. Some filesystems like e.g. UBIFS support LZ4, but now also support Zstd, because both are suitable for the task (and LZ4 was around earlier).

In the end it is a classic space vs. time trade-off and there is AFAIK no generic right or wrong answer (except that some algorithms are too slow to even be considered).

Re: LZ4 – Extremely fast compression

#96

Post by pottertheotter »

maxpert wrote: post #58

I've personally used LZ4 on production a scale that really proves how using a compression like LZ4 is more efficient than uncompressed data https://doordash.engineering/2019/01/02/speeding-up-redis-wi...

I've been using it reliably for the past several years for government documents needed for academic research. ~20mm documents that range from very small to 10s of MB. Each one is compressed using LZ4 and, overall, it's ~10% of the uncompressed size. Compressed it's about 2TB of data. It's unbelievable how fast they decompress.

Re: LZ4 – Extremely fast compression

#97

Post by specialist »

zX41ZdbW wrote: post #45

It is very interesting that compression libraries from Yann Collet outperform their Google counterparts by all means: lz4 >> snappy zstd >> brotli

Maybe Adjacent: Corecursive episode w/ Daniel Lemire was just terrific.

https://corecursive.com/frontiers-of-performance-with-daniel...

Some people are just really good at performance sensitive stuff.

Re: LZ4 – Extremely fast compression

#98

Post by OldHand2018 »

diggernet wrote: post #84
0-_-0 wrote: post #74

Earlier quoted context omitted.

Would that be like... a reverse fork? A defork? A korf?

Oh heck, let's just call it a spoon.

Nice. Really, anything that had solid support would be nice. My company delivers data to customers and our customers use all sorts of operating systems. Plain Zip seems to be our lowest common denominator because we can’t count on our customers being tech-savvy or having an IT department.

I really, really, really wish there were more compression standards that were supported on a wide variety of platforms. “Obscure” software package A just doesn’t cut it.

Re: LZ4 – Extremely fast compression

#99

Post by zrav »

roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

I've spent an afternoon testing zstd's custom dictionaries. It really only provides benefits on small data blocks. According to my tests, the largest blocks at which custom dictionaries could still provide a benefit is 8K, above that the compression ratio advantage compared to the default is definitely gone.

Re: LZ4 – Extremely fast compression

#100

Post by jgalt212 »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

> Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

I always assume S3 storage was compressed on the fly by AWS regardless of how the client chooses to store his/her data.

Post reply on HN