This is awesome since compression in parallel has been largely neglected in practice.
Many times faster (de)compression using multiple processors.
21–25 of 25 posts
Re: Many times faster (de)compression using multiple processors.
#22This is awesome since compression in parallel has been largely neglected in practice.
It's not neglected, it's just hard and a lot of projects failed. But there are many papers on the subject in the last 10 years. Also parallel is a vague term, it can mean multicore, SIMD, GPU and clusters. All different, with a unique set of tradeoffs.
Re: Many times faster (de)compression using multiple processors.
#23I like to use PPMd (via 7zip) for large volumes of text, but it seems to only be single-threaded, which is a shame. It cuts a good 30% again off the size of the .xml.bz2's that Wikipedia provides.
Re: Many times faster (de)compression using multiple processors.
#24It's good people get interested in the subject. But this is very odd and has some errors. For example xz requires a lot more memory resources than bzip2 (see benchmarks below, Mem column). http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/uiq/ Matt Mahoney mantains the best benchmarks on text and generic compression. Some of the best on the field (like Matt) usually hang out at encode.ru.
That benchmark runs xz with the -9e flags, which turn on its slowest and most memory-intensive mode. If you pass it -0 it only needs 3 MB to compress and 1 MB to decompress.
I usually use -0 with xz because it is extremely fast and memory-efficient yet still compresses better than gzip or bzip2. You can also use -0e for a slower, better compression that still requires only 1 MB to decompress. This way the decompressor can run entirely within the CPU's cache.
Re: Many times faster (de)compression using multiple processors.
#25It's good people get interested in the subject. But this is very odd and has some errors. For example xz requires a lot more memory resources than bzip2 (see benchmarks below, Mem column). http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/uiq/ Matt Mahoney mantains the best benchmarks on text and generic compression. Some of the best on the field (like Matt) usually hang out at encode.ru.
That benchmark runs xz with the -9e flags, which turn on its slowest and most memory-intensive mode. If you pass it -0 it only needs 3 MB to compress and 1 MB to decompress. I usually use -0 with xz because it is extremely fast and memory-efficient yet still compresses better than gzip or bzip2. You can also use -0e for a slower, better compression that still requires only 1 MB to decompress. This way the decompresso…
Yes, it's in the benchmarks. But comparing against bzip2 is not very meaningful. It's more relevant to compare with compressors in the same efficiency rate. For those flags it compresses to 26MB, in that range there are many equivalent ROLZ/LZP engines with similar numbers. For example csc32 is a bit faster but uses more memory.
http://mattmahoney.net/dc/text.html#2118
http://mattmahoney.net/dc/text.html#2300
Proper analysis would need benchmarking with different data and different flags for all compressors.