Compression Benchmarks: brotli, gzip, xz, bz2
1–10 of 20 posts
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#2There are standard corpuses for compression / decompression. Against a specific README file isn't interesting.
Give me the full results against the standard corpuses (in different sizes).
The biggest fallacy of "compression results" is non-standard data sets.
Edit: spent years in image/video compression under Langdon, involved with JPEG2000, etc.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#3Would have been useful to do this against Snappy and LZ4.
Both are quiet popular in the Big Data space.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#4> Hence we can conclude
By testing on one file. Furthermore, Brotli "cheats" by having a static dictionary including english words, so it doesn't make sense to compare against general-purpose compressors on the COPYING file.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#5There are standard corpuses for compression / decompression. Against a specific README file isn't interesting. Give me the full results against the standard corpuses (in different sizes). The biggest fallacy of "compression results" is non-standard data sets. Edit: spent years in image/video compression under Langdon, involved with JPEG2000, etc.
Do you know a current good comparison?
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#6There are standard corpuses for compression / decompression. Against a specific README file isn't interesting. Give me the full results against the standard corpuses (in different sizes). The biggest fallacy of "compression results" is non-standard data sets. Edit: spent years in image/video compression under Langdon, involved with JPEG2000, etc.
Do you know a current good comparison?
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#7The published results aren't consistent with the general understanding that xz is faster than bzip2 while compressing better even at the lowest settings.
Xz beyond the lowest compression setting quickly enters the realm of diminishing returns, but when even the lowest setting compresses better than bzip2 and is faster in both compression and decompression, there is no reason to use bzip2.
The opencpu post completely ignores the compression setting dimension, presenting an incomplete picture. Xz is shown as consistently the slowest by a wide margin, even relative to bzip2. This is unexpected, xz will be faster in both compression and decompression than bzip2 when using a compression setting appropriate for comparison with bzip2 compression ratios.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#8> Hence we can conclude By testing on one file. Furthermore, Brotli "cheats" by having a static dictionary including english words, so it doesn't make sense to compare against general-purpose compressors on the COPYING file.
Its job is to compress html, so at the very least the html-related dictionary contents make a lot of sense.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#9Would have been useful to do this against Snappy and LZ4. Both are quiet popular in the Big Data space.
Brotli is a "strong" compression algorithm so it's obviously going to be much slower with much better compression than "light" algorithms like Snappy or LZ4.
Re: Compression Benchmarks: brotli, gzip, xz, bz2
#10The published results aren't consistent with the general understanding that xz is faster than bzip2 while compressing better even at the lowest settings. Xz beyond the lowest compression setting quickly enters the realm of diminishing returns, but when even the lowest setting compresses better than bzip2 and is faster in both compression and decompression, there is no reason to use bzip2. The opencpu post completely…
Also xz upstream defaults to single threaded. However xz decompression scales pretty much linearly at least up to 4 cores - I wrote a parallel pxzcat to prove that: