Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

11–20 of 115 posts

Re: LZ4 – Extremely fast compression

#11

Post by wolf550e »

baybal2 wrote: post #9
moonchild wrote: post #4

Earlier quoted context omitted.

> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/

LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well. zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.

Google snappy is same class as lzo and lz4, not same class as brotli and zstd.

Re: LZ4 – Extremely fast compression

#14

Post by ignoramous »

wolf550e wrote: post #11
baybal2 wrote: post #9

Earlier quoted context omitted.

LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well. zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.

Google snappy is same class as lzo and lz4, not same class as brotli and zstd.

Also see Daniel Reiter Horn's DivANS built at Dropbox: https://dropbox.tech/infrastructure/building-better-compress...

Re: LZ4 – Extremely fast compression

#15

Post by jakozaur »

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.

Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor.

In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

Re: LZ4 – Extremely fast compression

#16

Post by [deleted] »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

[deleted]

Re: LZ4 – Extremely fast compression

#17

Post by maeln »

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

Re: LZ4 – Extremely fast compression

#18

Post by contravariant »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

If it's fast enough you wouldn't only be able to feed more data you'd be able to access it faster as well since you need less bandwidth.

That said it can be quite tricky to rewrite something to efficiently work on compressed data.

Re: LZ4 – Extremely fast compression

#19

Post by xianwen »

tpetry wrote: post #13
bombcar wrote: post #6

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...

I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?
Post reply on HN