Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

81–90 of 115 posts

Re: LZ4 – Extremely fast compression

#81

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

Assuming you have a larger record size than block size (with media files you probably want a 1M record size), and that you probably have ashift set to 12 (4k) or 13 (8k), then I believe you need to enable compression in order to prevent the final record from using the full record size. IOW, ZFS will pad out the final record with zeros to the full record size, then use compression on it (if enabled) so that the zeros don't need to be written to disk.

This article refers to the padding on the final record as "slack space" and states that you need to enable compression to eliminate the slack space.

https://arstechnica.com/information-technology/2020/05/zfs-1...

See also:

https://old.reddit.com/r/zfs/comments/gzcysy/h264_h265_media...

Re: LZ4 – Extremely fast compression

#82
post #48
post #19

Earlier quoted context omitted.

I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?

I've used both Btrfs and Zfs as Linux root filesystems and at the time I tested (about 4-5 years ago) Btrfs had much worse performance. I've heard that Btrfs greatly improved performance on recent kernels though. What bothers me about Zfs is that it uses a different caching mechanism (ARC) than Linux page cache. With ARC you actually see the memory used in tools like htop and gnome system monitor (it is not cool seei…

My main problem with ZFS is the very limited number of ways you can change your setup. No removing drives, no shrinking, etc. Probably fine for (bare-metal) production systems, but not so friendly with desktops/laptops, where I would still love to have snapshots and send-recv support.

Re: LZ4 – Extremely fast compression

#86
post #67

Earlier quoted context omitted.

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

I compression not on by default nowdays? Anyhow, I would not run ZFS with compression disabled completely. There are edgecases where you want it. The meta data? I can't remember the details. At least active the compression that just compresses zeros.

Since you can control compression settings per dataset, I'd just use a separate dataset for the videos directory with compression disabled.

Re: LZ4 – Extremely fast compression

#87
post #30
post #10

Earlier quoted context omitted.

I also appreciate LZ4's simplicity and tiny code footprint. zstd is brilliant as well, but in terms of code base it's a whole other beast.

Yes decompression on baremetal cortex m4 is a mere hundreds of bytes, you can decompress it from flash directly to its output buffer.

I've used it in bootloaders that have slow transfer mechanisms (uart, i2c) to get whatever speedup I can for a few hundred bytes of binary.

Re: LZ4 – Extremely fast compression

#88
post #6

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

For (low) bandwidth metrics yes, for any kind of latency sensitive workload not really.

The extra decompression on top of the data fetch latency can be quite noticeable. Sometimes that can be offset if the compression ratio is affecting a hitrate, and thereby decreasing the latency. The problem of course is that even with 10M IOP storage devices frequently it is really latency and an inability to keep 100k requests outstanding that limit perf to one's IO turnaround latency.

Put another way, compressed ram and disk are really popular in systems which are RAM constrained, or bandwidth limited because the cost of fetching 2x the data vs 1x and decompressing it is a win (think phones with emmc). The problem is that this doesn't really make sense on high end NVMe (or for that matter desktops/servers with a lot of RAM) where the cost of fetching 8k vs 4k is very nearly identical because the entire cost is front loaded on the initial few bytes, and after than the transfer overhead is minimal. Its even hard to justify on reasonable HD/RAID systems too for bandwidth tests since any IO that requires a seek by all the disks will then tend to flood the interface. AKA it takes tens of ms for the first byte, but then the rest of it comes in at a few GB/sec and decompressing at faster rates takes more than a single core.

edit: And to add another dimension, if the workload is already CPU bound, then the additional CPU overhead of compress/decompress in the IO path will likely cause a noticeable hit too. I guess what a lot of people don't understand is that a lot of modern storage systems are already compressed at the "hardware" layer by FTL's/etc.

Re: LZ4 – Extremely fast compression

#89
post #75
post #6

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

Predictability, no? Sometimes you want to know how large your data really is if it was to get onto an uncompressed filesystem.

You can use "du -A" to show the uncompressed size.

Re: LZ4 – Extremely fast compression

#90
post #77

A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image. I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data…

> lz4 (...) probably quite suitable for life filesystems and network protocols Actually, no. lz4 is less suitable than zstd for filesystems. BTW, lz4 is present in many mozilla tools like thunderbird: it's represented by its bastard child lz4json, which is diverging by just the headers don't work with regular lz4 tools > achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my Yo…

> Actually, no. lz4 is less suitable than zstd for filesystems.

Why's that? What benefit would I get from switching? Is it workload-dependent?

EDIT: To be clear, I'm not disagreeing; if zstd will work better, I want to know about it so that I can switch my pools to use it.

Post reply on HN