Show HN: pzip- blazing fast concurrent zip archiver and extractor
1–10 of 18 posts
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#2How does this compare against pigz? [1]. Afaik pigz comes bundled in some modern distros, I’ve also personally used it in some backup operations reliably
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#3Can't find out if this supports encryption, or streaming from stdin and/or to stdout. I haven't found one zip tool that does the above.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#4How does this compare against pigz? [1]. Afaik pigz comes bundled in some modern distros, I’ve also personally used it in some backup operations reliably [1]: https://zlib.net/pigz/
This appears to be for the .zip format, not gzip.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#5Can't find out if this supports encryption, or streaming from stdin and/or to stdout. I haven't found one zip tool that does the above.
Info-ZIP (aka zip(1)) does, assuming that a heavily flawed standard encryption and a single file input from stdin suffices.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#6Note that even when not considering the speedup due to the compression happening in multiple threads, the libraries used for compression here use much less CPU (user 3m33s) than "the standard zip utility" (user 13m13s i.e. 3.7 times the former -- if I understand correctly, this "standard" is Info-ZIP) which is a little less surprising knowing that the source for the later hasn't been updated for 15 years, while, if I understand correctly, this new go version depends on the compression routines maintained in https://pkg.go.dev/compress/flate
I also don't see the comparison of the resulting compression sizes of the two programs.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#7Note that even when not considering the speedup due to the compression happening in multiple threads, the libraries used for compression here use much less CPU (user 3m33s) than "the standard zip utility" (user 13m13s i.e. 3.7 times the former -- if I understand correctly, this "standard" is Info-ZIP) which is a little less surprising knowing that the source for the later hasn't been updated for 15 years, while, if I…
Yep the standard I refer to is Info-ZIP (zip(1)).
I will add the resulting compression sizes- there is not much between them (pzip was around 2% larger for the 10GB directory). Although, I do have some optimizations in mind which will bring this down further.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#8Can't find out if this supports encryption, or streaming from stdin and/or to stdout. I haven't found one zip tool that does the above.
At the moment it doesn't- if it's something you're looking for, please open an issue.
I didn't want to go down the route of implementing a bunch of features without there being a actual need for them, especially for an initial version.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#9Note that even when not considering the speedup due to the compression happening in multiple threads, the libraries used for compression here use much less CPU (user 3m33s) than "the standard zip utility" (user 13m13s i.e. 3.7 times the former -- if I understand correctly, this "standard" is Info-ZIP) which is a little less surprising knowing that the source for the later hasn't been updated for 15 years, while, if I…
This also depends on whether you distro is using zlib or zlib-ng, which is significantly faster.
Re: Show HN: pzip- blazing fast concurrent zip archiver and extractor
#10Note that even when not considering the speedup due to the compression happening in multiple threads, the libraries used for compression here use much less CPU (user 3m33s) than "the standard zip utility" (user 13m13s i.e. 3.7 times the former -- if I understand correctly, this "standard" is Info-ZIP) which is a little less surprising knowing that the source for the later hasn't been updated for 15 years, while, if I…
This also depends on whether you distro is using zlib or zlib-ng, which is significantly faster.
Do you know which distributions are using zlib-ng for zip and unzip programs?
If I understand, the improvement would be around 3 times less CPU use for the comparable resulting size, but I see it here shown for "minizip" not zip: