Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pigz: A parallel implementation of gzip for multi-core machines

github.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: pigz: A parallel implementation of gzip for multi-core machines

#61

Post by michaelbarton »

jiggawatts wrote: post #57

Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…

While the topic of compressing FASTQ comes up, you might be interested to know that kmer sorting FASTQ files can lead to around an 8% improvement in compression depending on the diversity.

I believe this is because it puts similar strings together in the tree gzip uses. clumpify.sh from bbtools is one example.

Re: pigz: A parallel implementation of gzip for multi-core machines

#62

Post by mgerdts »

xfalcox wrote: post #30

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation.

A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.

Re: pigz: A parallel implementation of gzip for multi-core machines

#63

Post by mgerdts »

necovek wrote: post #2

Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?

Back in the day I wrote this about how it improved Solaris kernel zone suspend:

https://web.archive.org/web/20160313033123/https://blogs.ora...

Re: pigz: A parallel implementation of gzip for multi-core machines

#64

Post by jiggawatts »

paisleyrob wrote: post #59
jiggawatts wrote: post #57

Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…

zstd has --adapt: --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed man…

I really should have read the documentation! That feature looks awesome, but in a quick test it could only use about 50% of the available output bandwidth. My upload speed is 50 Mbps, but zstd could only send about 25 Mbps.

Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.

Re: pigz: A parallel implementation of gzip for multi-core machines

#66

Post by chlorion »

jaimehrubiks wrote: post #15
dividuum wrote: post #13

Earlier quoted context omitted.

Maybe I’m missing something, but why send the tar generated stream through a non-compressing compressor when you could just send the tar directly?

I didn't have the tar, I created it using: tar --use-compress-program="pigz -0" ...

I think you are a little confused here.

Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression:

    tar -cf $directory.tar $directory
You can also pipe the created archive to stdout instead of to a file if you want to:

    tar -c $directory | wc -c
    # send to a remote system
    tar -c $directory | ssh dd of=~/$directory.tar
You can actually compress the archive by just piping it to a compression command:

    tar -c $directory | zstd --stdout > $directory.tar.zst
The above pipeline is probably very similar to what tar is doing internally when you pass "--use-compression-program".

So in your case using "pigz -0" is totally useless since tar creates an uncompressed archive by default. You can totally omit the "--use-compression-command" flag to do what you want.

Re: pigz: A parallel implementation of gzip for multi-core machines

#67

Post by gww »

There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files.

[1] https://github.com/samtools/htslib

Re: pigz: A parallel implementation of gzip for multi-core machines

#68

Post by jonnycomputer »

dspillett wrote: post #31
walrus01 wrote: post #27

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-...

pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block…

I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?
Post reply on HN