Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zstandard – Fast and efficient compression algorithm

github.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Zstandard – Fast and efficient compression algorithm

#11

Post by MichaelGG »

ot wrote: post #4

This looks very interesting. Before everybody starts making the obvious comparisons, note that this is from the same guy that made LZ4, and he is someone who clearly knows what he's doing. I've been following his work for some time. It looks like this is the evolution of Zhuff, an experimental (closed-source) compressor [1]. It is basically LZ4 followed by a fast entropy coder, specifically FSE [2], that is a flavor…

I wish some of these provided static, preset dictionaries. I'm working on a packet capture system for SIP, which has a similar layout as HTTP. Thus all the common header fields and values are perfect for a preset dictionary, and in fact, the compression doesn't even need to keep any more state than that dictionary. That is, packets share more with the preset dictionary than with each other.

LZ4 allows you to "prime the stream" as it were, but I'm not sure it is really made for this scenario. As far as I can tell, I'd have to essentially have separate compression/decompression calls for each packet, resetting the state to the dictionary between each packet.

Re: Zstandard – Fast and efficient compression algorithm

#13

Post by bhouston »

MichaelGG wrote: post #11
ot wrote: post #4

This looks very interesting. Before everybody starts making the obvious comparisons, note that this is from the same guy that made LZ4, and he is someone who clearly knows what he's doing. I've been following his work for some time. It looks like this is the evolution of Zhuff, an experimental (closed-source) compressor [1]. It is basically LZ4 followed by a fast entropy coder, specifically FSE [2], that is a flavor…

I wish some of these provided static, preset dictionaries. I'm working on a packet capture system for SIP, which has a similar layout as HTTP. Thus all the common header fields and values are perfect for a preset dictionary, and in fact, the compression doesn't even need to keep any more state than that dictionary. That is, packets share more with the preset dictionary than with each other. LZ4 allows you to "prime t…

What one needs is a preset dictionary. Have a hash of it during compression and then require a preset dic on decompress that has the same hash. Should be a straightforward extension but it is a format change. I think preset dic would be useful in a lot of contexts.

Re: Zstandard – Fast and efficient compression algorithm

#14

Post by MichaelGG »

bhouston wrote: post #13
MichaelGG wrote: post #11

Earlier quoted context omitted.

I wish some of these provided static, preset dictionaries. I'm working on a packet capture system for SIP, which has a similar layout as HTTP. Thus all the common header fields and values are perfect for a preset dictionary, and in fact, the compression doesn't even need to keep any more state than that dictionary. That is, packets share more with the preset dictionary than with each other. LZ4 allows you to "prime t…

What one needs is a preset dictionary. Have a hash of it during compression and then require a preset dic on decompress that has the same hash. Should be a straightforward extension but it is a format change. I think preset dic would be useful in a lot of contexts.

But the preset dic functionality I've seen, in say, LZ4, really is just the saved state of a normal compression run. So once you start compressing, the dictionary eventually evaporates as more data does in and backreferences can no longer point back that far into the dictionary. That's fine if all your content compresses well, but if the dictionary is a far superior state...

Re: Zstandard – Fast and efficient compression algorithm

#15

Post by MichaelGG »

bhouston wrote: post #12

How does it compare to lzham? That alto has some amazing speed and ratio and is zlib compatible.

This looks to compress much faster and suited for some realtime usage. lzham is OK for decompression (but still slower) and is like 8x slower on compression.

Re: Zstandard – Fast and efficient compression algorithm

#17

Post by tandemstrong »

MichaelGG wrote: post #11
ot wrote: post #4

This looks very interesting. Before everybody starts making the obvious comparisons, note that this is from the same guy that made LZ4, and he is someone who clearly knows what he's doing. I've been following his work for some time. It looks like this is the evolution of Zhuff, an experimental (closed-source) compressor [1]. It is basically LZ4 followed by a fast entropy coder, specifically FSE [2], that is a flavor…

I wish some of these provided static, preset dictionaries. I'm working on a packet capture system for SIP, which has a similar layout as HTTP. Thus all the common header fields and values are perfect for a preset dictionary, and in fact, the compression doesn't even need to keep any more state than that dictionary. That is, packets share more with the preset dictionary than with each other. LZ4 allows you to "prime t…

LZ4 can do that.

There is a function, called LZ4_decompress_safe_usingDict() which seems to match your objectives.

In case of doubt, you should ask directly the author, at : https://groups.google.com/forum/#!forum/lz4c

Re: Zstandard – Fast and efficient compression algorithm

#18

Post by jheriko »

the interface is good. :)

i can not praise that element enough. so many libraries have gigantic, needlessly elaborate interfaces, and forget to provide clean interfaces for the most common use cases. that makes it hard to use them... this however is easy to use.

Re: Zstandard – Fast and efficient compression algorithm

#19

Post by Lagged2Death »

ot wrote: post #5
X-Istence wrote: post #3

What's the Weissman score for this?

Do we really need to have this comment for any submission that mentions compression?

The Weissman score comment's Weissman score would be improved by merely linking to an earlier instance of the Weissman score comment.

Re: Zstandard – Fast and efficient compression algorithm

#20

Post by mkorfmann »

corysama wrote: post #8
maemre wrote: post #7

Earlier quoted context omitted.

I guess so. It is also the top comment on r/programming for this (link also contains the corresponding blog entry of the author): https://www.reddit.com/r/programming/comments/2tibrh/zstd_a_...

Please note that the HN community takes a rather strict approach when moderating comments that contribute noise to the conversation. "Nice article!" comments are routinely downvoted. As is sarcasm, witticisms, memes, references and other styles of comments that occur frequently but do not contribute to the discussion. It's a knowingly doomed attempt to hold back the flood of noise that covers Reddit.

Why so serious?
Post reply on HN