Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

bzip3

github.com

Post reply on HN
Print view View original

11–20 of 129 posts

Re: bzip3

#11

Post by dist-epoch »

myrmidon wrote: post #10
dist-epoch wrote: post #5

Earlier quoted context omitted.

with zstd at level 16 with default params (dict size, ...). Serious compression starts at level 19 and with much higher dict sizes. how is this an honest benchmark: bzip3 ... 12178M memory zstd ... 687M memory

There is a comparison with "zstd -19" on the Silesia corpus, showing better compression ratio for bzip3 (47.2 vs 53MB) while being ~5 times faster (and using only half the memory). Even if the examples are highly cherry-picked, it is quite suprising to me that such pareto-dominance is possible at all. edit: Tested it myself and found that it often also does slightly worse than zstd -19 in compression ratio but faster…

having used zstd, it has terrible defaults, optimized for speed and low-memory. You need to change it's params (not just level and dict size) to get high performance.

probably somebody should use a coding agent to do auto-research to optimize params for each compression algo, while matching one fixed goal - time, memory or size

Re: bzip3

#12

Post by ot »

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.

If we match the window size of zstd to that of bzip3 we get dramatically different results:

    % gzcat *.gz | time zstd -T8 -16 | wc -c  # baseline
     2819113884
    zstd -T8 -16  2054.50s user 3.47s system 783% cpu 4:22.80 total

    % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
     196405076
    zstd -T8 -16 --long=29  1083.06s user 2.41s system 783% cpu 2:18.55 total
Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).

(the baseline number is slightly different because I don't have the exact Perl version set used by the author)

Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.

Re: bzip3

#13

Post by m000 »

adrian_b wrote: post #6
kosolam wrote: post #2

Impressive compression benchmark. Four times smaller than z standard.

I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it. Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better…

Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.

Re: bzip3

#14

Post by altairprime »

sedatk wrote: post #7

The latest release is a year ago, the last commit is two months ago, and the build is failing. The claim “stronger than bzip2” is strange. What does it even mean? Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.

“for fairness, the benchmarks have been performed using single thread mode” (2025) https://news.ycombinator.com/item?id=42902241

Re: bzip3

#15

Post by CodesInChaos »

I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.

Re: bzip3

#16

Post by adrian_b »

m000 wrote: post #13
adrian_b wrote: post #6

Earlier quoted context omitted.

I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it. Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better…

Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.

I think so.

But developing the heuristics for choosing the appropriate compression algorithm for a stream of data is likely to need a very long time for compression tests of a lot of diverse training data, similarly to the training of a specialized ML model that classifies patterns.

Such heuristics should provide not only algorithm selection, but also parameter selection, when given only some simple input, e.g. the relative importances of compression ratio, decompression speed and compression speed.

Re: bzip3

#17

Post by altairprime »

CodesInChaos wrote: post #15

I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.

“Additional benchmarks on the same dataset” (2025) explores various compression levels with and without long range mode: https://news.ycombinator.com/item?id=42901476

Re: bzip3

#18

Post by amelius »

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.

Sounds like perhaps a nice testcase for formalization + AI?

Re: bzip3

#19

Post by CJefferson »

ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

Re: bzip3

#20

Post by finaard »

sedatk wrote: post #7

The latest release is a year ago, the last commit is two months ago, and the build is failing. The claim “stronger than bzip2” is strange. What does it even mean? Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.

Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page.

pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP support for standard bzip2 archives.

I'll need to try that next time I'm working with large archives - I learned about the pbzip2 limitations the hard way last time I was shuffling around a few multi-10GB archives, and was trying to speed things up fully utilising my 32 core threadripper.

Post reply on HN