Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#71

Post by linsomniac »

linsomniac wrote: post #68
sounds wrote: post #57

Earlier quoted context omitted.

Can you test it out and post results back here?

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

One more note:

"bzip3 -e -j 6 -b 50": 25 seconds, 125MB

So nearly as good as the best of xz, but in a 20th the time.

However: Do note that any unexpected use is met with a SIGSEGV: using as a filter, using "-j6" instead of "-j 6", not specifying "-e"...

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#72

Post by grumpyprole »

hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

There's more to a compression standard than benchmarks and ratios. Xz does not seem to score well in other, perhaps more important areas: https://www.nongnu.org/lzip/xz_inadequate.html

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#73

Post by bayindirh »

adrianmonk wrote: post #69
bayindirh wrote: post #44

Earlier quoted context omitted.

> I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this. I think it boils down to the feelings of the author (of the previous format). I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just…

> I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Is zstd actually an homage to zip? I'm not saying that it definitely isn't, but the only connection I know of myself is that they both begin with the letter Z, and the letter Z has a long association with data compression that goes back before the zip format / pkzip program. The LZ77, LZ78[1], and LZW[2] algorithms all predate the zip format. As do two…

> Is zstd actually an homage to zip?

I think so. Or more generally, DEFLATE family of algorithms used by zlib and ZIP.

> I'm not saying that it definitely isn't, but the only connection I know of myself is that they both begin with the letter Z, and the letter Z has a long association with data compression that goes back before the zip format / pkzip program.

> The LZ77, LZ78[1], and LZW[2] algorithms all predate the zip format.

I'm well aware of the Ziv, Lempel & Welch's work, since I've developed a compression algorithm [*] and did extensive research on the family before, however the usage of extension is a rather new information for me.

On the other side, official repository of zstd [0] lists zlib and other libraries which use DEFLATE in one way other, pointing back to Ziv's ZIP at the same time.

At the end of the day, it might be tipping its hat directly to "ZIP" per se, but to general direction of DEFLATE which is used by ZIP format too.

[*]: The algorithm I developed was working on syllables rather than bytes. It had a deterministic and fast hyphenation engine for the language, and used embedded dictionary to minimize bit-flip damage during transit. I have paper published from it, and still planning to re-implement and open it, since the PoC was beyond bad from a code quality perspective.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#74

Post by palaiologos »

linsomniac wrote: post #71
linsomniac wrote: post #68

Earlier quoted context omitted.

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

One more note: "bzip3 -e -j 6 -b 50": 25 seconds, 125MB So nearly as good as the best of xz, but in a 20th the time. However: Do note that any unexpected use is met with a SIGSEGV: using as a filter, using "-j6" instead of "-j 6", not specifying "-e"...

lies. not specifying -e displays an error message:

  % bzip3 -e -j 6 -b 50 corpus/calgary.tar
  % bzip3 -j 6 -b 50 corpus/calgary.tar
  bzip3 - A better and stronger spiritual successor to bzip2.
  Copyright (C) by Kamila Szewczyk, 2022. Licensed under the terms of GPLv3.
  Usage: bzip3 [-e/-d/-t/-c] [-b block_size] input output
  Operations:
    -e: encode
    -d: decode
    -t: test
  Extra flags:
    -c: force reading/writing from standard streams
    -b N: set block size in MiB
    -j N: set the amount of parallel threads
you can use bzip3 as a filter:

  % cat corpus/calgary.tar | bzip3 -b 10 -e -c | wc -c
  807959
and using "-j6" is simply being unable to read the help page.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#75

Post by bayindirh »

eru wrote: post #50
bayindirh wrote: post #44

Earlier quoted context omitted.

> I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this. I think it boils down to the feelings of the author (of the previous format). I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just…

> Open source software is designed to be mangled, modified, shared and leapfrogged. I agree in spirit, but I can also see why someone might want their source to be free and mangleable, but still care about trademarks. (Just imagine Linus Torvalds getting lots of emails with support requests for a hypothetical Linux2 operating system that I wrote, and that he has no relation with. That could become pretty annoying; ev…

A trademark changes the situation in every aspect. If your code is getting big, famous, and needs its name and likeness (e.g. Firefox), you get a trademark, and your derivatives shall not use the name. It's simple and clear.

When I quickly looked, bzip2 didn't have a trademark, and I assumed the developer don't care.

It's same for me. If I care, I'd trademark it, and prevent people from using it. If I'm giving the name away, I'd not trademark it. It's plain and simple.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#76

Post by JoshTriplett »

linsomniac wrote: post #68
sounds wrote: post #57

Earlier quoted context omitted.

Can you test it out and post results back here?

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

If you ramp up the compression level on zstd, does it get smaller than bzip3 before it gets to the point of taking more time?

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#77

Post by codewiz »

Matumio wrote: post #64
sounds wrote: post #57

Earlier quoted context omitted.

Can you test it out and post results back here?

No, but it would be nice to see visually where it is on the size vs (de-)compression speed pareto-front. Like this graphic (from the zstd homepage): https://raw.githubusercontent.com/facebook/zstd/master/doc/i...

This chart is from 2016. Both zstd and brotli are under active development, so I'd like to see a more recent comparison presented in this same format.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#78

Post by adgjlsfhk1 »

linsomniac wrote: post #68
sounds wrote: post #57

Earlier quoted context omitted.

Can you test it out and post results back here?

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

Can you also give decompression speeds?

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#80

Post by nousermane »

hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

> benchmarks don't compare to zstd.

  wget http://corpus.canterbury.ac.nz/resources/calgary.tar.gz
  zcat calgary.tar.gz|time zstd -19|wc -c

  902963 (=902.9KB, vs. 807.9KB for bzip3)
> "Better than bzip2" isn't exactly a convincing sales pitch

Sure, but nobody is pitching that. TFA does comapre to lzma (~xz), and claims bzip3 outperforms it quite handsomely in speed, while being competitive in compression ratio.

Post reply on HN