Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

41–50 of 115 posts

Re: LZ4 – Extremely fast compression

#41

Post by gens »

maeln wrote: post #17

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

It is used, and most every other compression technique was/is used in video games.

I was thinking of using LZ4, but it doesn't really work that great on floating point, and images are already compressed (png, jpg, and even BCn, can't be compressed much further). So idk. Good thing about lz4 is that it's very simple[0] and probably faster then memcpy().

http://ticki.github.io/blog/how-lz4-works/

Re: LZ4 – Extremely fast compression

#43

Post by kbumsik »

ksec wrote: post #35
kbumsik wrote: post #29

Earlier quoted context omitted.

Yes. Much faster. Especially for HDDs. But at a cost of a lot of RAM. Also lz4 compression can speed up your HDDs up to 10x (!) to read and 3x to write. [1, see "To compress, or not to compress, that is the question" section.] But it's going to have a considerably higher CPU usage as well. [1]: https://calomel.org/zfs_raid_speed_capacity.html

>But it's going to have a considerably higher CPU usage as well. I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15. Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )

I am not sure about cheap ARM devices but I am using an old Haswell i5-4670 and it is more than enough. So it won't be issue later.

Also, when you are talking about consumer NAS, the real problem is that any low-end systems can saturate the gigabit network (100MB/s) very easily so investing on extra resources for ZFS doesn't make difference. At least a 10Gbe network (which is beyond the average consumer) is required to actually make it useful.

Re: LZ4 – Extremely fast compression

#44

Post by zX41ZdbW »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

Using LZ4 can easily improve performance even if all data reside in memory.

This is the case in ClickHouse: if data is compressed, we decompress it in blocks that fit in CPU cache and then perform data processing inside cache; if data is uncompressed, larger amount of data is read from memory.

Strictly speaking, LZ4 data decompression (typically 3 GB/sec) is slower than memcpy (typically 12 GB/sec). But when using e.g. 128 CPU cores, LZ4 decompression will scale up to memory bandwidth (typically 150 GB/sec) as well as memcpy. And memcpy is wasting more memory bandwidth by reading uncompressed data while LZ4 decompression reads compressed data.

Re: LZ4 – Extremely fast compression

#47

Post by adzm »

jansan wrote: post #26

The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.

LZO is also GPL which can be a conflict, compared with the BSD license of LZ4.

Re: LZ4 – Extremely fast compression

#48

Post by tarruda »

xianwen wrote: post #19
tpetry wrote: post #13

Earlier quoted context omitted.

Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...

I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?

I've used both Btrfs and Zfs as Linux root filesystems and at the time I tested (about 4-5 years ago) Btrfs had much worse performance. I've heard that Btrfs greatly improved performance on recent kernels though.

What bothers me about Zfs is that it uses a different caching mechanism (ARC) than Linux page cache. With ARC you actually see the memory used in tools like htop and gnome system monitor (it is not cool seeing half your memory being used when no programs are running). ARC is supposed to release memory when needed (never tested though), so it might not be an issue.

After about an year of playing with both filesystems on my Linux laptop, I decided the checksumming is not worth the performance loss and switched back to ext4, which is significantly faster than both filesystems. Still use ZFS on backup drives for checksumming data at rest and easy incremental replication with `zfs send`.

Re: LZ4 – Extremely fast compression

#49

Post by hyperpape »

moonchild wrote: post #4
roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/

That table shows zstd comparing poorly at its faster settings, but at slower settings, it offers a significantly better compression ratio, albeit 3x slower decompression.

Re: LZ4 – Extremely fast compression

#50

Post by garaetjjte »

gens wrote: post #41
maeln wrote: post #17

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

It is used, and most every other compression technique was/is used in video games. I was thinking of using LZ4, but it doesn't really work that great on floating point, and images are already compressed (png, jpg, and even BCn, can't be compressed much further). So idk. Good thing about lz4 is that it's very simple[0] and probably faster then memcpy(). http://ticki.github.io/blog/how-lz4-works/

>and even BCn, can't be compressed much further

S3TC is block compression, so if there is repeating data in images it will compress quite well.

Post reply on HN