How difficult is this new standard going to be to implement in another language? It seems highly sophisticated -- which is great, of course -- but the cost of that is relying on giants like Facebook to maintain their One True Implementation. For software this is (usually) fine; for a nee standard, it's a problem.
Smaller and faster data compression with Zstandard
21–30 of 166 posts
Re: Smaller and faster data compression with Zstandard
#22[deleted]
Re: Smaller and faster data compression with Zstandard
#23Looks very interesting, however I'm not impressed by the name. "Zstandard"??? With ".zstd" as the extension? I don't like it. They should have named it letter -zip, along the lines of gzip, bzip, and xzip, with the extension letter z. "fz" would have been a good one since they work at Facebook.
Agreed. How did they not call it "Pied Piper"?
I was waiting for that reference.
Re: Smaller and faster data compression with Zstandard
#24How difficult is this new standard going to be to implement in another language? It seems highly sophisticated -- which is great, of course -- but the cost of that is relying on giants like Facebook to maintain their One True Implementation. For software this is (usually) fine; for a nee standard, it's a problem.
The format itself is documented (https://github.com/facebook/zstd/blob/master/zstd_compressio...) with the intention of other implementations and language bindings being readily available. We also have a zlib-compatible API for easier porting to applications already using Zlib. Our hope is that Zstandard is both easy to use and easy to contribute to.
Re: Smaller and faster data compression with Zstandard
#25The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn'…
The README at https://github.com/facebook/zstd mentions brotli
Re: Smaller and faster data compression with Zstandard
#26Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstandard, but I may be able to answer some as well; we both work on compression at Facebook.
Re: Smaller and faster data compression with Zstandard
#27I have been waiting for this to hit 1.0 and more importantly get popular so that I can use it everywhere. I am really a fan of Yann Collet's work. These are extremely impressive work specially when you consider that lz4 seems to be better than snappy (by google) and zstandard from LZFSE (from apple). I think he is the first one to write a practical fast arithmetic coder using ANS. And look at how his huffman implementation blazes past zlib huffman though compresses less than FSE [0]. I also like reading his blog posts. While a lot of them goes over my head I can generally make a sense of what he is trying and why something's working despite the complexity.
Re: Smaller and faster data compression with Zstandard
#28The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn'…
Brotli is available in this graph - https://github.com/facebook/zstd/blob/master/images/DCspeed5...
This seems to be slightly old doc - from April.
Re: Smaller and faster data compression with Zstandard
#29There is just so much awesome stuff in this article. Finite State Entropy and Asymmetric Numeral System are completely new concepts to me (I've got 7 open tabs just from references FB supplied in the article), as is repcode modeling. I love that they've already built in granular control over the compression tradeoffs you can make, and I can't wait to look into Huff0. If anyone outside of Facebook has started playing…
I think you should look at the author's blog [0]. There is lots of in depth explanation of not just what he is doing but how he got there, the failed steps, the intermediate steps.
Re: Smaller and faster data compression with Zstandard
#30There is just so much awesome stuff in this article. Finite State Entropy and Asymmetric Numeral System are completely new concepts to me (I've got 7 open tabs just from references FB supplied in the article), as is repcode modeling. I love that they've already built in granular control over the compression tradeoffs you can make, and I can't wait to look into Huff0. If anyone outside of Facebook has started playing…
Indeed ANS is difficult: it is the biggest innovation in compression in the last 20 years. Its author has some nice but dense slides about it.
https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf
Not sure exactly when repcodes were invented. Igor Pavlov has already used them in 7zip.