For maximum compression, pLzip offers lzma compression in parallel: https://www.nongnu.org/lzip/plzip.html
pigz: A parallel implementation of gzip for multi-core machines
81–90 of 107 posts
Re: pigz: A parallel implementation of gzip for multi-core machines
#82Earlier quoted context omitted.
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code chu…
Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27... So interesting if they implemented and tested that and get only a marginal CRC speedup. edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739
igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.
Re: pigz: A parallel implementation of gzip for multi-core machines
#83Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
Apart from tuning up the compression you may gain a lot by clustering reads in FASTQ files using clumpify from BBMap:
https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb...
You may also filter out duplicates and depending on what you do correct base errors or get rid of low complexity reads.
Re: pigz: A parallel implementation of gzip for multi-core machines
#84Earlier quoted context omitted.
Given that, why wouldn't this just be upstreamed into gzip? If it's a clean, simple solution that's just expanding the use of a technique that's already in the core binary?
> Given that, why wouldn't this just be upstreamed into gzip? I suspect to keep the standard gzip as simple, small, and stable, as possible. It does the job, does it well enough, with minimal dependencies, has done for many years, and can do so in a wide array of systems including very small environments (in part due to the minimal dependencies). Core tools like that typically don't get major updates, just security &…
While I'm in agreement on all of those points; I find adoption of new tools extremely difficult. The are modern alternatives to many commands; ls, cat, grep etc but if they're not "the default" it becomes near impossible to switch to them.
Given almost all desktops/servers/mobile phones are likely to be multi core these days, if gzip gained multithreading on desktop it could save time and energy for the whole planet potentially, that seems like a worthwhile benefit?
Re: pigz: A parallel implementation of gzip for multi-core machines
#85Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files.
The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.
Re: pigz: A parallel implementation of gzip for multi-core machines
#86If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.…
on the other hand, bzip2 is pretty much obsoleted now by xzip
Also see: http://enwp.org/zstd
parallel-friendly, trades off compression level for speed
Re: pigz: A parallel implementation of gzip for multi-core machines
#87Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
> one quirk of that space is that they work with enormous text files.
Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed
Re: pigz: A parallel implementation of gzip for multi-core machines
#88Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files. The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.
These days, skill is not about how you use tools in IT, but simply knowing that a tool exists.
Re: pigz: A parallel implementation of gzip for multi-core machines
#89Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
> one quirk of that space is that they work with enormous text files. Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed
Because scientists are woefully undereducated on the finer points of computer science. Does it (eventually) produce publishable papers? is all they care about.
Re: pigz: A parallel implementation of gzip for multi-core machines
#90I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.
Oh, that's the pun. I just saw Parallel Implementation of GZip...