Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

141–150 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#141

Post by pmarreck »

muragekibicho wrote: post #133
pmarreck wrote: post #117

Earlier quoted context omitted.

Basically, `zstd --train ` will output a dictionary file, and then the `-D ` option when used for either compression or decompression will then use that dictionary first. You can also investigate "man zstd" or google "zstd --train" for more details. The directory for the training must consist of many small files each of which is an example artifact; if you want to split, say, a single log file into files of each line…

Thank you so much. I was trying to create a dictionary last night and your comment was sent by God. You're doing the Lord's work frfr! I followed you on GitHub!

remember that if you don't understand a particular line of code, you can have chatgpt explain it... have fun

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#142

Post by PaulHoule »

bick_nyers wrote: post #38
adamgordonbell wrote: post #30

Earlier quoted context omitted.

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…

In video games long loading times for levels is a serious pain point, so video game developers put a lot of effort into tuning up compression algorithms to get the best wall clock time considering both the time to fetch content from storage and the time to decompress.

If the target is a console you may know exactly what hardware is there so you can justify the effort in tuning. (it’s more complex today because you have a choice of what kind of storage to use with your XBOX). With a PC or phone your results may vary a lot more.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#143

Post by jltsiren »

daneel_w wrote: post #121
capableweb wrote: post #58

Earlier quoted context omitted.

> There literally are no trade-offs; it is better in every objective way. There literally are trade-offs, you started your comment describing one of them. If you want as wide out-of-the-box support as possible, you'd go with gzip. The Compression Streams browser API only supports gzip (+ deflate) so if you wanna compress something natively in the browser without 3rd party libraries (or slow JS implementation), gzip s…

People had your exact sentiments, concerns and hesitations after gzip showed up in the early 90s. Eventually they moved on from pkzip/lzh/etc. to better, modern software - some on their own owed to being reasonable people, and some being dragged along with claws in ground while screaming about "breaking support".

It was easier back then, because there were fewer people developing new compressors.

I've lost count how many compressors have been marketed as a replacement for gzip over the decades. And it's always a replacement for gzip. Every time a new compressor starts getting popular, people start promoting a new even better replacement, and gzip never gets properly replaced.

zstd finally has some potential to replace gzip, but only if people accept it's good enough and stop trying to replace it with something even better.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#144

Post by powturbo »

adamgordonbell wrote: post #109
dralley wrote: post #76

Earlier quoted context omitted.

Fastest open source compression algorithms. RAD game tools have proprietary ones that are faster and have better compression ratios, but since you have to pay for a license, they will never be widespread.

Interesting. Are there any benchmarks you can share? On their website they only compare decompression speed and with zlib and LZMA. It would be interesting to compare to LZ4 HC mode, that unity uses.

Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors.

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#145

Post by wanttocomment »

fnordpiglet wrote: post #94
wanttocomment wrote: post #71

Earlier quoted context omitted.

I agree with most of these points except blaming this on processes vs threads. The only difference is all memory being shared by default, vs explicitly deciding what memory to share. With all the emphasis on memory safety on HN you think this point would be appreciated.

That’s fairly new, with threads and processes becoming basically the same. Historically threads didn’t exist, then they were horrifically implemented and non standard, then they standardized and were horrifically implemented, then they were better implemented but the APIs were difficult to use safely, etc etc. Also threads were much more light weight than a process. This shifted with light weight processes, etc.

That's true, but if you're looking that far back, multicore is new too

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#146

Post by daneel_w »

jltsiren wrote: post #143
daneel_w wrote: post #121

Earlier quoted context omitted.

People had your exact sentiments, concerns and hesitations after gzip showed up in the early 90s. Eventually they moved on from pkzip/lzh/etc. to better, modern software - some on their own owed to being reasonable people, and some being dragged along with claws in ground while screaming about "breaking support".

It was easier back then, because there were fewer people developing new compressors. I've lost count how many compressors have been marketed as a replacement for gzip over the decades. And it's always a replacement for gzip. Every time a new compressor starts getting popular, people start promoting a new even better replacement, and gzip never gets properly replaced. zstd finally has some potential to replace gzip, b…

"zstd finally has some potential to replace gzip"

bzip2 and xz have had the potential to replace gzip for the vast majority of users and use cases since more than a decade - and in many cases they have.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#147

Post by bick_nyers »

PaulHoule wrote: post #142
bick_nyers wrote: post #38

Earlier quoted context omitted.

People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…

In video games long loading times for levels is a serious pain point, so video game developers put a lot of effort into tuning up compression algorithms to get the best wall clock time considering both the time to fetch content from storage and the time to decompress. If the target is a console you may know exactly what hardware is there so you can justify the effort in tuning. (it’s more complex today because you ha…

Don't most games/game engines use TGA format for their textures? Those are all RLE-encoded if I'm not mistaken (which is very fast but very inefficient space-wise). Or perhaps that is just at game creation and those will get baked to some other image format for distribution?

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#148

Post by Retr0id »

mgerdts wrote: post #130
Retr0id wrote: post #125

Earlier quoted context omitted.

> the effective rate hits 12 GiB/s I assume this is for decompressing multiple independent deflate streams in parallel? What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)

> I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy You saw this, right? https://news.ycombinator.com/item?id=35915285

I consider using an index to be "cheating" - or rather, my intended use-case is decompression of a stream that you've never seen before, which was generated by a "dumb" compressor.

That said, the approach I intend to take is similar. The idea is that one thread is dedicated to "looking ahead", parsing as fast as it can (or even jumping far ahead and using heuristics to re-sync the parse state. There will be false-positives but you can verify them later), building an index but not actually doing decompression, while secondary threads are spawned to do decompression from the identified block start points. The hard part is dealing with missing LZ references to data that hasn't yet been decompressed. Worst-case performance will be abysmal, but I think on most real-world data, you'll be able to beat a serial decompressor if you can throw enough threads at it.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#149

Post by aidenn0 »

lathiat wrote: post #22
aigoochamna wrote: post #19

Earlier quoted context omitted.

How big is that file... I have 2TB files compressed down to ~300GB and gunzip'ing them takes ~2-3 hours. Granted, that's still a long ass time, but not 2-3 days. If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P

bzip2 is much slower than gzip.

bzip2 is not much slower at compression than gzip; gzip is far faster at decompression though. Either way since bzip2 is a block based compressor, parallelization is trivial, and parallel implementations started appearing about 20 years ago; pbzip2 is almost certainly in whichever package manager is in use for TFA.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#150

Post by aidenn0 »

pizza234 wrote: post #3
Twirrim wrote: post #2

Similarly, for bzip2 there is pbzip2 ( http://compression.great-site.net/pbzip2/?i=1 ). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.

AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.

bzip2 is already block based though, so there is no compatibility issue in that specific case (vs pbzip2) though I think pbzip2 supports larger blocks than the original bzip2.
Post reply on HN