Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LZ4 – Extremely fast compression

lz4.github.io

Post reply on HN
Print view View original

51–60 of 115 posts

Re: LZ4 – Extremely fast compression

#51

Post by ksec »

kbumsik wrote: post #43
ksec wrote: post #35

Earlier quoted context omitted.

>But it's going to have a considerably higher CPU usage as well. I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15. Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )

I am not sure about cheap ARM devices but I am using an old Haswell i5-4670 and it is more than enough. So it won't be issue later. Also, when you are talking about consumer NAS, the real problem is that any low-end systems can saturate the gigabit network (100MB/s) very easily so investing on extra resources for ZFS doesn't make difference. At least a 10Gbe network (which is beyond the average consumer) is required…

> can saturate the gigabit network

Yes I completely forgotten about that. But 2.5/5 Gbps Ethernet is finally coming along.

Hopefully someday. ZFS will come.

Re: LZ4 – Extremely fast compression

#52

Post by mrec »

contravariant wrote: post #18
jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

If it's fast enough you wouldn't only be able to feed more data you'd be able to access it faster as well since you need less bandwidth. That said it can be quite tricky to rewrite something to efficiently work on compressed data.

GPU texture compression formats would be the obvious example. With the additional constraint that they need to support efficient random access.

Re: LZ4 – Extremely fast compression

#54

Post by rav »

Jonnax wrote: post #22
AnthonBerg wrote: post #2

Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/ (Linking to this more for the overview than the Windows tool in itself.)

The frustrating thing about 7zip is that whilst it's opensource, a single author just does code drops. So there's a bunch of forks with useful features that'll never be adopted because there's no collaboration. At least that's what I could tell when I looked into it

It sounds like it needs a fork to combine all the forks, similar to what Neovim did as a fork of Vim.

Re: LZ4 – Extremely fast compression

#55

Post by gens »

garaetjjte wrote: post #50
gens wrote: post #41

Earlier quoted context omitted.

It is used, and most every other compression technique was/is used in video games. I was thinking of using LZ4, but it doesn't really work that great on floating point, and images are already compressed (png, jpg, and even BCn, can't be compressed much further). So idk. Good thing about lz4 is that it's very simple[0] and probably faster then memcpy(). http://ticki.github.io/blog/how-lz4-works/

>and even BCn, can't be compressed much further S3TC is block compression, so if there is repeating data in images it will compress quite well.

I tried compressing BC7 (from AMD-s Compressonator) using lz4c and it wasn't much.

Just re-ran it (with -hc, version 1.9.3(latest now)) and: "Compressed 5592544 bytes into 5413783 bytes ==> 96.80%".

7z with "7z a -mx=9 -t7z asd.7z albedo_PNG_BC7_1.KTX" does 5592544 bytes to 4844707 bytes (down to ~87%).

Original file is 10227047 bytes (PNG, RGBA), i can't remember if the ktx has mipmaps.

EDIT: Note that the image is fairly noisy (gravel). Could/should be better with more "artificial" textures.

I don't know if ktx does some extra compression, but, looking at it, i doubt it.

PS I think that BC could be massaged at compression to be better compressible, and i think i read something about that. Don't remember.

Re: LZ4 – Extremely fast compression

#56

Post by stavros »

tpetry wrote: post #13
bombcar wrote: post #6

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...

Do you mean LZ4? I fear your comment is a bit misleading as-is.

Re: LZ4 – Extremely fast compression

#57

Post by YetAnotherNick »

unsigner wrote: post #38
roncohen wrote: post #3

another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

Zstd is very different - it includes an entropy coder. LZ4 only finds repeated matches, but then doesn't encode them very efficiently. To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly, while Zstd will compress it 8:1 converging to an encoding where a '1' bit is A, and a '0' bit is B.

> To put it simplistically, if you have a file which is a (good) random mix of an equal number A and B characters, LZ4 won't be able to compress it significantly

I checked it. LZ4 is still reducing the size to half, no idea why half. So for 10 MB file it compresses to 5 MB.

Edit: checked with highest compression and it compresses 1MB file to 185KB. So what the parent wrote is false.

Re: LZ4 – Extremely fast compression

#59

Post by KitDuncan »

jakozaur wrote: post #15

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is qu…

I have a ZFS pool with exclusively video files. Probably won't see any benefit in enabling LZ4 there right?

Re: LZ4 – Extremely fast compression

#60

Post by bob1029 »

maeln wrote: post #17

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

If you really want to impress your customers, use SQLite to aggregate LZ4-compressed entities. In many AAA games, there can be hundreds of thousands of assets to load & keep track of. If you have to load an entire scene from disk, you could write a simple SQL query to select all assets assigned to the scene (i.e. a SceneAssets mapping table) and then stream them all into memory from the unified database file handle.
Post reply on HN