Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller and faster data compression with Zstandard

code.facebook.com

Post reply on HN
Print view View original

71–80 of 166 posts

Re: Smaller and faster data compression with Zstandard

#71

Post by jacobolus »

espadrine wrote: post #69

The following link points to a fairly good benchmark / tool that showcases the tradeoffs in real life: since (de)compression takes time, what is the fastest way to transmit data at a given transfer speed? https://quixdb.github.io/squash-benchmark/unstable/#transfer... Spoilers: zstd wins at ethernet and wifi (and is among the best in 4G), lz4 wins at hard drive encryption… both were designed by the same author.

Charles Bloom’s Oodle new codecs handily beat them across the scale:

http://cbloomrants.blogspot.com http://www.cbloom.com/oodle_arm_report/ http://www.cbloom.com/rants.html

Caveat: proprietary commercial product.

Re: Smaller and faster data compression with Zstandard

#75

Post by wolf550e »

cmurphycode wrote: post #48
cmurphycode wrote: post #40

Earlier quoted context omitted.

There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the…

Imgur album now compares brotli and zstd. Looks like zstd is always significantly faster, but Brotli compresses slightly better for every dataset except the XML one. This is with brotli level 1, by the way. My understanding is that brotli is pretty quick through the first few levels, but the levels that ask for the highest compression are insanely slow (which is a valuable thing to have as an option, for things like…

For "compress once, decompress many times" workloads like game assets, you might want to check out Oodle, a proprietary library that kills zstd in decompression speed while achieving better compression ratios.

http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me...

Re: Smaller and faster data compression with Zstandard

#76

Post by tmd83 »

mark-r wrote: post #47
Jerry2 wrote: post #33

Earlier quoted context omitted.

Apple's goals were also to have a low-energy de/compressor suitable for mobile. I'd love to see some comparisons of the two of them running on ARM.

I'd love to see that too. Remember that if the processor uses 2x the power but gets done 3x faster, it's still 1.5x more efficient overall.

Thats what I was thinking. I haven't found any validation or even the rationale behind lzfse's supposed lower power usage. I can think of two things.

1. Apple tried to write a fast and reasonably compressible version of LZ4 thus improving power usage by creating LZFSE since none existed but beaten out handsomely by Zstandard.

2. Following a parent comment, Zstandard might some of the things that are dependent on a highly OoO cpu with lots of caches, extremely good branch predictor that could be significantly slower on an ARM even the apple one despite how good they are. Or they could still be slower but on ARM the gap might not be as big and the decision not as cut and dry as it seems now.

Would love to know what the actual case is from someone involved in LZFSE.

Re: Smaller and faster data compression with Zstandard

#77

Post by Someone »

jdcarter wrote: post #38
markonen wrote: post #12

The goals sound similar to Apple's LZFSE (see https://github.com/lzfse/lzfse for more). Any comparison out there?

That was my first thought, too. I installed and ran both against a tar'd set of PDF files totaling 435MB in size. My timings: lzfse 45 MB/s encode, 229 MB/s decode, 1.12 comp ratio zstd 181 MB/s encode, 713 MB/s decode, 1.13 comp ratio The numbers are so dramatically different that I ran several different tests, but those results showed the same rough results. I used default command-line options for both tools, and b…

Can you repeat those tests using the library shipping with Mac OS X?

I'm asking because Apple claims the github code is a reference implementation. Those typically aren't tuned for speed or efficiency.

Re: Smaller and faster data compression with Zstandard

#78

Post by pbarnes_1 »

jdcarter wrote: post #38
markonen wrote: post #12

The goals sound similar to Apple's LZFSE (see https://github.com/lzfse/lzfse for more). Any comparison out there?

That was my first thought, too. I installed and ran both against a tar'd set of PDF files totaling 435MB in size. My timings: lzfse 45 MB/s encode, 229 MB/s decode, 1.12 comp ratio zstd 181 MB/s encode, 713 MB/s decode, 1.13 comp ratio The numbers are so dramatically different that I ran several different tests, but those results showed the same rough results. I used default command-line options for both tools, and b…

Fast == power efficient, though.

Not surprising. Apple-originated tech is always kind of middle of the road.

Re: Smaller and faster data compression with Zstandard

#79

Post by ryao »

This is an awesome blog post that is very well written, but the lack of incompressible performance analysis prevents It from providing a complete overview of zstd.

Incompressible performance measurements are important for interactive/realtime workloads and the numbers are extremely interesting because they can differ dramatically from the average case measurements. LZ4 for instance has been measured at doing 10GB/sec on incompressible data on a single core of a modern Intel Xeon processor. At the other end of the spectrum is the worst case scenario for incompressible data where performance slows to a crawl. I do not recall any examples in this area, but the point is that it is possible for algorithms to have great average case performance and terrible worst case performance. Quick sort is probably the most famous example of that concept.

I have no reason to suspect that zstd has bad incompressible performance, but the omission of incompressible performance numbers is unfortunate.

Post reply on HN