Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pigz: A parallel implementation of gzip for multi-core machines

github.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: pigz: A parallel implementation of gzip for multi-core machines

#101

Post by mort96 »

dima_vm wrote: post #38
omoikane wrote: post #10

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).

No clue if this is the reason, but exceptions are actually really fast. With them (or longjmp-based error handling), the "happy path" gets to not even thing about errors. With error returns (be it Go's error interface, Rust's Result type, or C's "this function returns a negative number on error" pattern), you need a branch after every call which might fail, which does have a measurable (if usually small) impact on performance. Given pigz's entire raison d'etre is performance, it wouldn't surprise me if this impacted the choice of error handling style.

Re: pigz: A parallel implementation of gzip for multi-core machines

#102

Post by nextaccountic »

exyi wrote: post #95
nextaccountic wrote: post #87

Earlier quoted context omitted.

> one quirk of that space is that they work with enormous text files. Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed

It was probably fasta of fastq format, which is pretty much line separated strings of the DNA letters. BSON won't help with that. You could try to squeeze the each of the 4 letters into 2 bits, but they use few more letters to indicate "unknown", so it's not that easy. And even the simplest compression algorithms just do that for you. And text is easier to work with in any hacked up script. There are some binary form…

> And even the simplest compression algorithms just do that for you.

They do, but you need to uncompress to read the N-th letter

Re: pigz: A parallel implementation of gzip for multi-core machines

#103

Post by ROTMetro »

jiggawatts wrote: post #88
jakobnissen wrote: post #85

Earlier quoted context omitted.

Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files. The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.

These days, skill is not about how you use tools in IT, but simply knowing that a tool exists .

Hasn't that always been the case? When I lived in the Bay Area and networked, and was exposed to so much of how people did things, I was able to land much more contracts because I could come up with simple existing solutions. Once I moved away and wasn't exposed to cool people doing cool problem solving things, I had to switch into more bog standard consulting.

It blew people's minds when we couple implement huge projects that they could never get of the ground for years in just a matter of months because of 'this one cool trick'.

Man I miss Santa Cruz ಥ﹏ಥ worst mistake of my life to leave there.

Re: pigz: A parallel implementation of gzip for multi-core machines

#104

Post by pQd »

jiggawatts wrote: post #64
paisleyrob wrote: post #59

Earlier quoted context omitted.

zstd has --adapt: --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed man…

I really should have read the documentation! That feature looks awesome, but in a quick test it could only use about 50% of the available output bandwidth. My upload speed is 50 Mbps, but zstd could only send about 25 Mbps. Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.

if you're running your test over the internet [ fluctuating latency, some packet losses ] - try enabling BBR [1] tcp congestion control algorithm on the sender side to utilize the available bandwidth more efficiently.

[1] https://en.wikipedia.org/wiki/TCP_congestion_control#TCP_BBR

Re: pigz: A parallel implementation of gzip for multi-core machines

#105

Post by alias_neo »

dspillett wrote: post #91
alias_neo wrote: post #84

Earlier quoted context omitted.

While I'm in agreement on all of those points; I find adoption of new tools extremely difficult. The are modern alternatives to many commands; ls, cat, grep etc but if they're not "the default" it becomes near impossible to switch to them. Given almost all desktops/servers/mobile phones are likely to be multi core these days, if gzip gained multithreading on desktop it could save time and energy for the whole planet…

When reading from media with high random-access latency (for instance traditional hard drives) going parallel could make things much slower, and it will reduce the compression achieved (if only by a small amount), and it will increase the amount of memory consumed during the process, so I wouldn't want it to be the default. Nor would I particularly want it to try be clever and detect the usefulness of going parallel…

Interesting, I had no idea this would have such an effect, I simply assumed it would be performed in-memory and written out sequentially. I can agree with a flagged option being a nice idea, but what's so heinous about adding a dependency?

I try to avoid it generally in my coffee too, but for something that could potentially offer a measurable benefit in a world of multi core-solid state computing, would it not be "worth it"?

Re: pigz: A parallel implementation of gzip for multi-core machines

#106

Post by walrus01 »

jonnycomputer wrote: post #98
walrus01 wrote: post #76

Earlier quoted context omitted.

zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options: HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable…

Yes, indeed I had already read that in the man page, but I did not feel it answered my question, because in my mind, and state of naivety about compression, logically at least, a compression algorithm and final file format needn't stand in a 1-1 relation. But I suppose that ZLIB is just the DEFLATE algorithm, then?

yes, zlib is the library underpinning traditional gzip

zstandard's own zstandard-own-compression format is its own separate much newer thing.

zlib, for instance

https://packages.debian.org/source/buster/zlib

Re: pigz: A parallel implementation of gzip for multi-core machines

#107

Post by Xorlev »

evanelias wrote: post #24
Xorlev wrote: post #11

Earlier quoted context omitted.

Pretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds. I recall learning the technique from Tumblr's eng blog. https://engineering.tumblr.com/post/7658008285/efficiently-c...

I wrote that Tumblr eng blog post, glad to see it's still making the rounds! I later joined FB's mysql team a few years after that, although I can't quite remember if FB was still using pigz by that time. (also, hi Eric!) Separately, at Tumblr I vaguely remember examining some alternative to pigz that was consistently faster at the time (11 years ago) because pigz couldn't parallelize decompression. Can't quite remem…

Small world! Thanks for writing that, it was a really clever way to do it and saved me a bunch of time. :)
Post reply on HN