Awesome
Smaller and faster data compression with Zstandard
91–100 of 166 posts
Re: Smaller and faster data compression with Zstandard
#92Earlier quoted context omitted.
>what about the web The browser sends the server a request header indicating which compression methods it understands. Current Firefox for example sends Accept-Encoding: gzip, deflate, br meaning the server is free to send a response compressed with either gzip, deflate or brotli. Or the server can choose to send the data uncompressed. This means the adoption path for the web would be an implemtation in at least one…
Nice, so really browsers and servers can implement it independently. Seems elegant, I like that browsers can advertise multiple compression formats. Thanks for the info!
It's also possible to implement a decompressor in javascript to support browsers which don't do it natively. The performance would likely suck but if you're truly bandwidth constrained and don't mind users having a bit of a lag, it's an option...
Re: Smaller and faster data compression with Zstandard
#93I have been waiting for this to hit 1.0 and more importantly get popular so that I can use it everywhere. I am really a fan of Yann Collet's work. These are extremely impressive work specially when you consider that lz4 seems to be better than snappy (by google) and zstandard from LZFSE (from apple). I think he is the first one to write a practical fast arithmetic coder using ANS. And look at how his huffman implemen…
Cyan4973/Yann also is well known for xxhash[1], which is one of the faster hashers[2] out there. Post a new hasher and people will probably ask about xxhash (ex: metrohash[3]). Guy is an absolute machine. If you search Google for 'zstd' right now, you'll find him, not Facebook, namely: https://github.com/Cyan4973/zstd . Glad his work is being supported by someone now! Immensely well deserved, after so many years of h…
I found his (Yann's) blog posts the most helpful for getting a basic idea of what was going on with FSE:
https://fastcompression.blogspot.com/2013/12/finite-state-en...
and the followup
https://fastcompression.blogspot.fr/2014/01/fse-decoding-how...
Re: Smaller and faster data compression with Zstandard
#94Isn't it a bit presumptuous to call your own thing "standard"?
Like these guys: https://github.com/feross/standard, which is permissive to the point it's completely pointless. You can follow it to the letter and still end up with vomit code.
Re: Smaller and faster data compression with Zstandard
#95[deleted]
Re: Smaller and faster data compression with Zstandard
#96Beautiful.
Re: Smaller and faster data compression with Zstandard
#97Earlier quoted context omitted.
Apple's goals were also to have a low-energy de/compressor suitable for mobile. I'd love to see some comparisons of the two of them running on ARM.
Is low energy significantly different from high performance code? I thought the general advice was make the code as fast as possible so the work can be finished and the CPU slow/power down again. Are there cases where an algorithm takes 10x the wall clock time to execute, but actually uses less energy on the same chip? (Memory use/access is the main thing I guess that could be different.)
>> Is low energy significantly different from high performance code?
It can be. Certain operations are more power efficient than others - subtraction and then a check for negative value instead of comparison, certain vector operations, loop unrolling, using less memory/bus traffic...
>> Are there cases where an algorithm takes 10x the wall clock time to execute, but actually uses less energy on the same chip?
Slower code can be more power efficent. You're just tuning for different results.
Re: Smaller and faster data compression with Zstandard
#98This is an awesome blog post that is very well written, but the lack of incompressible performance analysis prevents It from providing a complete overview of zstd. Incompressible performance measurements are important for interactive/realtime workloads and the numbers are extremely interesting because they can differ dramatically from the average case measurements. LZ4 for instance has been measured at doing 10GB/sec…
zstd goes at > 1 GB/s on uncompressible data.
It has some fast heuristics for such cases too.
Re: Smaller and faster data compression with Zstandard
#99Earlier quoted context omitted.
Nice, so really browsers and servers can implement it independently. Seems elegant, I like that browsers can advertise multiple compression formats. Thanks for the info!
It's also possible to implement a decompressor in javascript to support browsers which don't do it natively. The performance would likely suck but if you're truly bandwidth constrained and don't mind users having a bit of a lag, it's an option...
Assuming the size of the decompressor isn't larger than the savings you gained from using this compression algo over another...
Re: Smaller and faster data compression with Zstandard
#100The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn'…
Slide 4 here has one: https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf