Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

101–110 of 115 posts

Re: LZ4 – Extremely fast compression

#101

Post by npteljes »

Mashimo wrote: post #67
KitDuncan wrote: post #59

Earlier quoted context omitted.

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

I compression not on by default nowdays? Anyhow, I would not run ZFS with compression disabled completely. There are edgecases where you want it. The meta data? I can't remember the details. At least active the compression that just compresses zeros.

I enabled it for my media storage, knowing that it wouldn't matter much, and indeed it doesn't. Compressratios are 1.00, 1.00 and 1.01.

Re: LZ4 – Extremely fast compression

#102

Post by johnisgood »

pmarreck wrote: post #71
tyingq wrote: post #68

Earlier quoted context omitted.

"Even storing items in-memory compressed sometimes is profitable" LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

TIL about zRAM, emailed myself this config to try at work https://www.techrepublic.com/article/how-to-enable-the-zram-...

Thanks, I just enabled it. We will see. :)

Re: LZ4 – Extremely fast compression

#103

Post by dgacmu »

PSeitz wrote: post #33

I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex

Has anyone written the appropriate Reader wrappers to use this with file io? (Asking b/c a quick search didn't turn anything up.)

Re: LZ4 – Extremely fast compression

#104

Post by mintyc »

https://blog.logrocket.com/rust-compression-libraries/

Although implementations arein rust, I assume the provided benchmarks are representative of any optimised implementation...

Many compressor algorithms are compared on several data sets.

The results tables show compressed size, compression and decompression times for a number of normal and pathological cases.

Get a good feel about strengths and weaknesses.

Some algs really go downhill in pathological cases, such as with random data.

Do consider encryption too though you probably want to do that on the compressed data set where possible.

Sometimes external encryption means you will be stuck with something close to pathological...

Re: LZ4 – Extremely fast compression

#105

Post by PSeitz »

dgacmu wrote: post #103
PSeitz wrote: post #33

I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex

Has anyone written the appropriate Reader wrappers to use this with file io? (Asking b/c a quick search didn't turn anything up.)

file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.

Re: LZ4 – Extremely fast compression

#106

Post by tomcam »

zamalek wrote: post #91
tyingq wrote: post #68

Earlier quoted context omitted.

"Even storing items in-memory compressed sometimes is profitable" LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

Microsoft found that compressed pages were always faster, because added by de-/compression was less than the latency to disk (given a sufficiently fast compression algorithm). As a bonus, it's also faster to read and write compressed pages to disk (if that absolutely has to happen). Zswap is therefore enabled by default on Windows. I configure my own kernel on Arch and Zswap is enabled by default there, too.

Pages as in the 8K (for example) data structure used to store portions of files on disk, or pages as in text files? I'm assuming the former but I am not very good with file system internals

Re: LZ4 – Extremely fast compression

#107

Post by pdimitar »

zX41ZdbW wrote: post #44
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

Using LZ4 can easily improve performance even if all data reside in memory. This is the case in ClickHouse: if data is compressed, we decompress it in blocks that fit in CPU cache and then perform data processing inside cache; if data is uncompressed, larger amount of data is read from memory. Strictly speaking, LZ4 data decompression (typically 3 GB/sec) is slower than memcpy (typically 12 GB/sec). But when using e.…

That's extremely interesting. Thanks for sharing.

I'm not into C/C++ for years though and now I wouldn't grok the code, sadly.

Re: LZ4 – Extremely fast compression

#108

Post by tyingq »

tomcam wrote: post #106
zamalek wrote: post #91

Earlier quoted context omitted.

Microsoft found that compressed pages were always faster, because added by de-/compression was less than the latency to disk (given a sufficiently fast compression algorithm). As a bonus, it's also faster to read and write compressed pages to disk (if that absolutely has to happen). Zswap is therefore enabled by default on Windows. I configure my own kernel on Arch and Zswap is enabled by default there, too.

Pages as in the 8K (for example) data structure used to store portions of files on disk, or pages as in text files? I'm assuming the former but I am not very good with file system internals

Memory pages, typically 4kb.

Re: LZ4 – Extremely fast compression

#109

Post by pdimitar »

kbumsik wrote: post #43
ksec wrote: post #35

Earlier quoted context omitted.

>But it's going to have a considerably higher CPU usage as well. I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15. Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )

I am not sure about cheap ARM devices but I am using an old Haswell i5-4670 and it is more than enough. So it won't be issue later. Also, when you are talking about consumer NAS, the real problem is that any low-end systems can saturate the gigabit network (100MB/s) very easily so investing on extra resources for ZFS doesn't make difference. At least a 10Gbe network (which is beyond the average consumer) is required…

I repurposed a micro Dell Optiplex 3060 with 8GB RAM and two external HDDs totalling 9TB of space. The CPU is an i3. The whole thing takes less space than a book.

I have lz4 enabled and the gigabit link is almost completely saturated when transferring: 119 MB/s out of the total theoretical 125.

No ZIL, no L2ARC devices are attached. That thing is _flying_ as a home NAS.

Re: LZ4 – Extremely fast compression

#110

Post by londons_explore »

zamalek wrote: post #91
tyingq wrote: post #68

Earlier quoted context omitted.

"Even storing items in-memory compressed sometimes is profitable" LZ4 is one of the algorithms supported by Zram in Linux. It's fairly popular for people using things like a Raspberry PI that have a smaller amount of RAM.

Microsoft found that compressed pages were always faster, because added by de-/compression was less than the latency to disk (given a sufficiently fast compression algorithm). As a bonus, it's also faster to read and write compressed pages to disk (if that absolutely has to happen). Zswap is therefore enabled by default on Windows. I configure my own kernel on Arch and Zswap is enabled by default there, too.

Microsoft's compressed pages implementation seems to work far better than zswap on Linux.

I can't quite see why - perhaps the logic to decide which pages to compress is different, or there is too much code in the swap subsystem that slows down the compression/decompression process...

Post reply on HN