Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3: A spiritual successor to BZip2

github.com

Post reply on HN
Print view View original

31–40 of 181 posts

Re: Bzip3: A spiritual successor to BZip2

#31

Post by [deleted] »

linsomniac wrote: post #19
linsomniac wrote: post #2

Highlight (benchmark of Perl source code): The results follow: xz -T16 -9 -k - 2'056'645'240 bytes (c=12m09s, d=4m40s) bzip2 -9 -k - 3'441'163'911 bytes (c=17m16s, d=9m22s) bzip3 -b 256 - 1'001'957'587 bytes (c=7m10s, d=4m6s? Unclear on source page) bzip3 -b 511 - 546'456'978 bytes (c=7m08s, d=4m6s? Unclear) zstd -T12 -16 - 3'076'143'660 bytes (c=6m32s, d=3m51s) edit: Adding times and compression levels

My standard test is compressing a "dd" disc image of a Linux install (I use these for work), with unused blocks being zeroed. Results: Uncompressed: 7,516,192,768 zstd: 1,100,323,366 bzip3 -b 511 -j 4: 1,115,125,019

[deleted]

Re: Bzip3: A spiritual successor to BZip2

#32

Post by loeg »

ThatGuyRaion wrote: post #28
loeg wrote: post #22

Earlier quoted context omitted.

zstd is faster and provides better compression than gzip at every point on the curve. There is no reason to use gzip these days other than backwards compatibility.

zstd is a great setup, for sure, but the build system they use is patently awful. Can someone make an autoconf or hell, even CMake build system for them pleasee???

Just install the binary package from whatever distro you use? Why do you need to build it?

But if it matters, FreeBSD has a pretty trivial BSDmake build of it:

https://github.com/freebsd/freebsd-src/blob/main/lib/libzstd...

https://github.com/freebsd/freebsd-src/blob/main/usr.bin/zst...

You could easily do something similar in GNU make or whatever without the dependencies on the FBSD build system. It's basically just the classic "cc -c a bunch of C files, then link them."

Re: Bzip3: A spiritual successor to BZip2

#33

Post by loeg »

chefandy wrote: post #30
hulitu wrote: post #9

> Bzip3: A better and stronger spiritual successor to BZip2 Please, no. What's next ? BZip4 and BZip5, each one incompatible with each other ?

Yeah I'm sick of this. Did you know you can't even use ext2/3/4 together on the same partition? What a mess.

The ext4/3 filesystems, notably, can read/write ext2 (and for ext4: ext3) filesystems in a compatible way.

Re: Bzip3: A spiritual successor to BZip2

#34

Post by wakawaka28 »

hulitu wrote: post #9

> Bzip3: A better and stronger spiritual successor to BZip2 Please, no. What's next ? BZip4 and BZip5, each one incompatible with each other ?

If they have to be incompatible then it's better to not conceal that. Generalized file formats require you to implement more stuff to support them, and we can't tell the format by looking at the file name.

Re: Bzip3: A spiritual successor to BZip2

#35

Post by zimpenfish »

linsomniac wrote: post #2

Highlight (benchmark of Perl source code): The results follow: xz -T16 -9 -k - 2'056'645'240 bytes (c=12m09s, d=4m40s) bzip2 -9 -k - 3'441'163'911 bytes (c=17m16s, d=9m22s) bzip3 -b 256 - 1'001'957'587 bytes (c=7m10s, d=4m6s? Unclear on source page) bzip3 -b 511 - 546'456'978 bytes (c=7m08s, d=4m6s? Unclear) zstd -T12 -16 - 3'076'143'660 bytes (c=6m32s, d=3m51s) edit: Adding times and compression levels

Did a test on a 1.3G text file (output of `find -f -printf ...`); Macbook Pro M3 Max 64GB. All timings are "real" seconds from bash's builtin `time`.

    files.txt         1439563776
    bzip2 -9 -k       1026805779 71.3% c=67 d=53
    zstd --long -19   1002759868 69.7% c=357 d=9
    xz -T16 -9 -k      993376236 69.0% c=93 d=9
    zstd -T12 -16      989246823 68.7% c=14 d=9
    bzip3 -b 256       975153650 67.7% c=174 d=187
    bzip3 -b 256 -j12  975153650 67.7% c=46 d=189
    bzip3 -b 511       974113769 67.6% c=172 d=187
    bzip3 -b 511 -j12  974113769 67.6% c=77s d=186
I'll stick with zstd for now (unless I need to compress the Perl source, I guess.)

(edited to add 12 thread runs of bzip3 and remove superfluous filenames)

Re: Bzip3: A spiritual successor to BZip2

#36

Post by Borg3 »

jgalt212 wrote: post #18

I poke around in this space periodically, but I've never found a compelling reason to move away from gzip.

Same here. Storage is not that expensive so I do NOT care to squize every byte out of archive. Also, im more into retro, so portability and memory usage is more importand for me :)

Re: Bzip3: A spiritual successor to BZip2

#37

Post by unnah »

mmastrac wrote: post #4

I've studied the Burrows-Wheeler Transform, I understand the transformation, I've re-implemented it countless times for kicks, I see how it improves compressability, but for the life of me the intuition of _why_ it works has never really clicked. It's a fantastic bit of algorithmic magic that will always impress me to see it.

The Burroughs-Wheeler transform has been described as a unique algorithm idea in that there are no non-trivial variations or related algorithms, unlike more conventional compression algorithms, which can be tweaked and improved in so many ways. There is no general compression theory in which BWT could be described as a special case.

It looks to me that the above still holds: Bzip2 and Bzip3 are simply combining more conventional compression algorithms with the BWT, which itself is still the same old transform. Bzip2 does Huffman coding after BWT, and Bzip3 does arithmetic coding.

Re: Bzip3: A spiritual successor to BZip2

#38

Post by zimpenfish »

loeg wrote: post #21
linsomniac wrote: post #2

Highlight (benchmark of Perl source code): The results follow: xz -T16 -9 -k - 2'056'645'240 bytes (c=12m09s, d=4m40s) bzip2 -9 -k - 3'441'163'911 bytes (c=17m16s, d=9m22s) bzip3 -b 256 - 1'001'957'587 bytes (c=7m10s, d=4m6s? Unclear on source page) bzip3 -b 511 - 546'456'978 bytes (c=7m08s, d=4m6s? Unclear) zstd -T12 -16 - 3'076'143'660 bytes (c=6m32s, d=3m51s) edit: Adding times and compression levels

Why -T12 for zstd and T16 for xz? How many threads is bzip3 using?

From the source, it looks like bzip3 defaults to 1 thread if not explicitly set by arguments.

Re: Bzip3: A spiritual successor to BZip2

#39

Post by lexicality »

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE.

I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for anything remotely serious is a good idea with that line in the readme.

Post reply on HN