Smaller and faster data compression with Zstandard • Hacker News Board

Re: Smaller and faster data compression with Zstandard

#161

Post by gliptic » Sat, Sep 03, 2016, 11:38 AM UTC

lifthrasiir wrote: post #125

gliptic wrote: post #124

Earlier quoted context omitted.

There are now actually open source decompressors[0] for Oodle formats, and yes, many use rANS. Especially LZNA's clever use of SSE2 to decode and update 16 symbol models is interesting. FSE (which is tANS) is a different beast, though, and more suitable for rarely updated models. [0] http://encode.ru/threads/2577-Open-source-Kraken-Mermaid-Sel...

...and got DMCAed [1]. At the very least the discussion itself is interesting for the inside infos, though. [1] https://github.com/github/dmca/blob/master/2016/2016-08-30-O...

That was because of the belief that he had used an illegal copy of the DLL, and that he had copied all implementation details (neither is true). A representative from RAD said they would withdraw the DMCA complaint now after that misunderstanding was cleared.

Re: Smaller and faster data compression with Zstandard

#162

Post by hasszhao » Sat, Sep 03, 2016, 7:33 PM UTC

Ok

Re: Smaller and faster data compression with Zstandard

#163

Post by sctb » Sat, Sep 03, 2016, 8:32 PM UTC

hasszhao wrote: post #162

Ok

Please don't do this, comments on HN need to be substantive.

Re: Smaller and faster data compression with Zstandard

#164

Post by mydeardiary » Sun, Sep 04, 2016, 11:45 AM UTC

If facebook hopes the new compression algorithm to be a standard, why doesn't it publish an IETF RFC draft? Will it follow OpenDNS way of dnscrypt by open-sourcing the reference implementation without publishing any IETF RFC draft?

Re: Smaller and faster data compression with Zstandard

#165

Post by twitch_checksum » Mon, Sep 05, 2016, 4:11 AM UTC

tmd83 wrote: post #34

ctur wrote: post #26

Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstandard, but I may be able to answer some as well; we both work on compression at Facebook.

I am really looking forward to this. I usually like to read more than vidoes but for complicated topic with a good presenter it can actually be a comprehensive starting point. Would the video also be posted today or we will have to wait? One thing I haven't figured out from either today's post or Yann's blog is whether Zstandard is switching between huff0 and FSE depending on compression level or is it somehow using…

Video is now available here : https://atscaleconference.com/videos/real-time-data-compress...

Re: Smaller and faster data compression with Zstandard

#166

Post by vintermann » Tue, Sep 06, 2016, 6:56 AM UTC

eln1 wrote: post #158

vintermann wrote: post #155

Earlier quoted context omitted.

With regular arithmetic coding and Huffman coding you don't need to send over a dictionary. Instead, you can have an adaptive model that learns to compress the data as it goes (e.g. keeps running track of symbol frequencies), and it will still be reversible. I thought this wasn't possible with ANS. Or has this changed?

It depends if you have static or adaptive coder. Static is much cheaper, uses the same probabilities for the entire data block (e.g. 30 kB), probabilities are stored in the header - practically all Huffman and tANS compressors (however, there are considered exceptions: https://en.wikipedia.org/wiki/Adaptive_Huffman_coding ). Adaptive can start with e.g. uniform probability (no need to store in header) and learns on t…

I know that's what adaptive coding means, I thought that was impossible with rANS. All the descriptions of rANS I could find (back when I looked at it) described it terms of a static model, and I ran into problems trying to generalize it to an adaptive one.

Thanks for the resources.