Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pigz: A parallel implementation of gzip for multi-core machines

github.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: pigz: A parallel implementation of gzip for multi-core machines

#91

Post by dspillett »

alias_neo wrote: post #84
dspillett wrote: post #58

Earlier quoted context omitted.

> Given that, why wouldn't this just be upstreamed into gzip? I suspect to keep the standard gzip as simple, small, and stable, as possible. It does the job, does it well enough, with minimal dependencies, has done for many years, and can do so in a wide array of systems including very small environments (in part due to the minimal dependencies). Core tools like that typically don't get major updates, just security &…

While I'm in agreement on all of those points; I find adoption of new tools extremely difficult. The are modern alternatives to many commands; ls, cat, grep etc but if they're not "the default" it becomes near impossible to switch to them. Given almost all desktops/servers/mobile phones are likely to be multi core these days, if gzip gained multithreading on desktop it could save time and energy for the whole planet…

When reading from media with high random-access latency (for instance traditional hard drives) going parallel could make things much slower, and it will reduce the compression achieved (if only by a small amount), and it will increase the amount of memory consumed during the process, so I wouldn't want it to be the default. Nor would I particularly want it to try be clever and detect the usefulness of going parallel as that could lead to unexpected inconsistency.

I think there is a case for including it as a selectable option, as with --rsyncable, unless this adds extra dependencies (pthreads was mentioned in other comments).

Re: pigz: A parallel implementation of gzip for multi-core machines

#92

Post by brigandish »

makapuf wrote: post #51
jallmann wrote: post #46

Earlier quoted context omitted.

Golang has panic / recover / defer which are functionally similar to exceptions. It's actually a fun exercise to implement a pseudo-syntax for try/catch/finally in terms of those primitives.

Go has exceptions but its definitely not advised to use those as an error mechanism. Recover is really a last chance effort for recovery, not a standard error catching method.

That's why they're called exceptions, because some errors are exceptional, otherwise they should be handled in the non-exceptional, standard flow of the program.

Re: pigz: A parallel implementation of gzip for multi-core machines

#94

Post by mytherin »

omoikane wrote: post #10

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

Postgres uses a similar custom implementation of try/catch [1].

[1] https://github.com/postgres/postgres/blob/master/src/include...

Re: pigz: A parallel implementation of gzip for multi-core machines

#95

Post by exyi »

nextaccountic wrote: post #87
jiggawatts wrote: post #57

Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…

> one quirk of that space is that they work with enormous text files. Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed

It was probably fasta of fastq format, which is pretty much line separated strings of the DNA letters. BSON won't help with that. You could try to squeeze the each of the 4 letters into 2 bits, but they use few more letters to indicate "unknown", so it's not that easy. And even the simplest compression algorithms just do that for you.

And text is easier to work with in any hacked up script.

There are some binary formats (BAM, I think), but people often prefer the text format anyway. When compressed, the size is pretty much the same

Re: pigz: A parallel implementation of gzip for multi-core machines

#96

Post by elmolino89 »

jakobnissen wrote: post #85
jiggawatts wrote: post #57

Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…

Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files. The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.

bgzf has a clear advantage when it comes to sorted BED/VCF/GTF formats, especially if one does index these. But frankly I have no idea if it may improve IO times for the fastq files when read by say bwa or another mapper. Do you have any experience with that? I have seen some mapping times improvement using fastqs with clustered reads (by clumpify).

Re: pigz: A parallel implementation of gzip for multi-core machines

#98

Post by jonnycomputer »

walrus01 wrote: post #76
jonnycomputer wrote: post #68

Earlier quoted context omitted.

I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?

zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options: HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable…

Yes, indeed I had already read that in the man page, but I did not feel it answered my question, because in my mind, and state of naivety about compression, logically at least, a compression algorithm and final file format needn't stand in a 1-1 relation. But I suppose that ZLIB is just the DEFLATE algorithm, then?

Re: pigz: A parallel implementation of gzip for multi-core machines

#99

Post by jakobnissen »

elmolino89 wrote: post #96
jakobnissen wrote: post #85

Earlier quoted context omitted.

Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files. The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.

bgzf has a clear advantage when it comes to sorted BED/VCF/GTF formats, especially if one does index these. But frankly I have no idea if it may improve IO times for the fastq files when read by say bwa or another mapper. Do you have any experience with that? I have seen some mapping times improvement using fastqs with clustered reads (by clumpify).

Yes, bgzf can be much faster in practice. Both because the underlying gzip implementation can be simplified (see e.g. libdeflate), and because it can be more effectively parallelised. It doesn't matter if it's IO-bound of course, but compression rarely is.

Re: pigz: A parallel implementation of gzip for multi-core machines

#100

Post by anistas »

chasil wrote: post #45
xfalcox wrote: post #30

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.

Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster) create small image of size 2MiB and below?

You won't see any improvement from parallelization on this type of data.

Post reply on HN