Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

81–90 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#81

Post by codewiz »

149764 wrote: post #38
palaiologos wrote: post #22

Earlier quoted context omitted.

zstd -19 linux.tar 462.58s user 0.76s system 100% cpu 217M memory 7:42.56 total % wc -c linux.tar.zst linux.bz3 134980904 linux.tar.zst 129255792 linux.bz3

# compression bzip3 -j 4 -e linux-5.18-rc6.tar linux-5.18-rc6.tar.bz3 user: 345.48s system: 0.59s cpu: 373% total: 1:32.75 zstd -19 --long -T4 -f linux-5.18-rc6.tar user: 1270.48s system: 0.89s cpu: 376% total: 5:37.9 > du -b linux-5.18-rc6.tar.* | sort -rn | reln 1.000000 130907738 linux-5.18-rc6.tar.zst 0.994715 130215881 linux-5.18-rc6.tar.bz3 With additional ‘--ultra -22’ tar.zst is smaller, but the compression t…

Is reln a command to add a column of relative numbers to the left? Neat.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#82

Post by the-alchemist »

I'm really interested in GPU-based compression / decompression.

Anyone know what the current SOTA GPU-based algorithms are, and why they haven't taken off?

Brotli has gotten browser support, so it seems to my naive self that a GPU-based algorithm is just waiting take over.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#83

Post by pcwalton »

The Burrows-Wheeler transform, which was the main innovation of bzip2 over gzip, and which this bzip3 retains, is one of the most fascinating algorithms to study: https://en.wikipedia.org/wiki/Burrows-Wheeler_transform

It hasn't been used lately because of the computational overhead, but it's interesting and I'm glad that there's still work in this area. For anyone interested in algorithms it's a great one to wrap your head around.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#84

Post by 149764 »

codewiz wrote: post #81
149764 wrote: post #38

Earlier quoted context omitted.

# compression bzip3 -j 4 -e linux-5.18-rc6.tar linux-5.18-rc6.tar.bz3 user: 345.48s system: 0.59s cpu: 373% total: 1:32.75 zstd -19 --long -T4 -f linux-5.18-rc6.tar user: 1270.48s system: 0.89s cpu: 376% total: 5:37.9 > du -b linux-5.18-rc6.tar.* | sort -rn | reln 1.000000 130907738 linux-5.18-rc6.tar.zst 0.994715 130215881 linux-5.18-rc6.tar.bz3 With additional ‘--ultra -22’ tar.zst is smaller, but the compression t…

Is reln a command to add a column of relative numbers to the left? Neat.

Yes, a small python script.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#85

Post by eesmith »

palaiologos wrote: post #74
linsomniac wrote: post #71

Earlier quoted context omitted.

One more note: "bzip3 -e -j 6 -b 50": 25 seconds, 125MB So nearly as good as the best of xz, but in a 20th the time. However: Do note that any unexpected use is met with a SIGSEGV: using as a filter, using "-j6" instead of "-j 6", not specifying "-e"...

lies. not specifying -e displays an error message: % bzip3 -e -j 6 -b 50 corpus/calgary.tar % bzip3 -j 6 -b 50 corpus/calgary.tar bzip3 - A better and stronger spiritual successor to bzip2. Copyright (C) by Kamila Szewczyk, 2022. Licensed under the terms of GPLv3. Usage: bzip3 [-e/-d/-t/-c] [-b block_size] input output Operations: -e: encode -d: decode -t: test Extra flags: -c: force reading/writing from standard str…

The code has, at https://github.com/kspalaiologos/bzip3/blob/bf2f0e02fd59f4c4... :

            } else if (argv[i][1] == 'j') {
                workers = atoi(argv[i + 1]);
                i++;
If the last argument is "-j6" then this will read past the end of the allocated argv strings and try to do atoi(NULL):

  % ./bzip3 -j3
"-j6" is standard getopt() behavior, and the default expected behavior from Unix/POSIX systems.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#86

Post by chasil »

Beltalowda wrote: post #28
baybal2 wrote: post #16

Earlier quoted context omitted.

1. zStandard is not a standard 2. Bzip2 is somewhat is a standard 3. zStandard is not a substitute for Bzip2

In what way is bzip2 more of a "standard" than zstd? bzip2 doesn't even seem to have any official reference description of its file format; just an "unofficial" one[1], whereas zstd is RFC 8478[2]. When I evaluated various compression algorithms a few years ago zstd came ahead of bzip2 in every metric. [1]: https://github.com/dsnet/compress/blob/master/doc/bzip2-form... [2]: https://datatracker.ietf.org/doc/html/rfc8…

That is interesting.

The author of lzip has harsh criticism of xz, and admiration of bzip2 for error detection/correction and "rightsizing" the container format.

I use lzip in preference to xz unless I need portability.

https://www.nongnu.org/lzip/xz_inadequate.html

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#87

Post by ttybird2 »

eesmith wrote: post #85
palaiologos wrote: post #74

Earlier quoted context omitted.

lies. not specifying -e displays an error message: % bzip3 -e -j 6 -b 50 corpus/calgary.tar % bzip3 -j 6 -b 50 corpus/calgary.tar bzip3 - A better and stronger spiritual successor to bzip2. Copyright (C) by Kamila Szewczyk, 2022. Licensed under the terms of GPLv3. Usage: bzip3 [-e/-d/-t/-c] [-b block_size] input output Operations: -e: encode -d: decode -t: test Extra flags: -c: force reading/writing from standard str…

The code has, at https://github.com/kspalaiologos/bzip3/blob/bf2f0e02fd59f4c4... : } else if (argv[i][1] == 'j') { workers = atoi(argv[i + 1]); i++; If the last argument is "-j6" then this will read past the end of the allocated argv strings and try to do atoi(NULL): % ./bzip3 -j3 "-j6" is standard getopt() behavior, and the default expected behavior from Unix/POSIX systems.

They are not disputing that - they actually acknowledged it. Instead they are disputing that omitting -e will lead to a crash.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#88

Post by Beltalowda »

chasil wrote: post #86
Beltalowda wrote: post #28

Earlier quoted context omitted.

In what way is bzip2 more of a "standard" than zstd? bzip2 doesn't even seem to have any official reference description of its file format; just an "unofficial" one[1], whereas zstd is RFC 8478[2]. When I evaluated various compression algorithms a few years ago zstd came ahead of bzip2 in every metric. [1]: https://github.com/dsnet/compress/blob/master/doc/bzip2-form... [2]: https://datatracker.ietf.org/doc/html/rfc8…

That is interesting. The author of lzip has harsh criticism of xz, and admiration of bzip2 for error detection/correction and "rightsizing" the container format. I use lzip in preference to xz unless I need portability. https://www.nongnu.org/lzip/xz_inadequate.html

As far as I know xz and zstd and completely unrelated?

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#89

Post by nousermane »

JoshTriplett wrote: post #76
linsomniac wrote: post #68

Earlier quoted context omitted.

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

If you ramp up the compression level on zstd, does it get smaller than bzip3 before it gets to the point of taking more time?

No, not even if you disregard time altogether. Compression of Calgary Corpus on a random old laptop:

             sec   KB
  gzip      0.19  1070
  zstd      0.02  1063
  zstd -19  1.47   897
  bzip2     0.35   891
  brotli    8.12   862
  xz        1.35   853
  bzip3     0.52   808

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#90

Post by loeg »

klodolph wrote: post #58
loeg wrote: post #53

Earlier quoted context omitted.

My impression was that lz4 ratios were still marginally better than zstd for the same compression speed, and decompression is much, much faster.

We must be looking at different graphs. See p.7 https://indico.fnal.gov/event/16264/contributions/36466/atta... You can see the classic Pareto frontier, with LZ4 filling the niche at the very bottom right edge of the graph.

I'm looking at http://facebook.github.io/zstd/ "Benchmarks."

P.7 of your slides doesn't seem to cover Zstd --fast or decompression. It would be interesting to see how Zstd performs in those modes.

Post reply on HN