Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#61

Post by adgjlsfhk1 »

klodolph wrote: post #58
loeg wrote: post #53

Earlier quoted context omitted.

My impression was that lz4 ratios were still marginally better than zstd for the same compression speed, and decompression is much, much faster.

We must be looking at different graphs. See p.7 https://indico.fnal.gov/event/16264/contributions/36466/atta... You can see the classic Pareto frontier, with LZ4 filling the niche at the very bottom right edge of the graph.

note that for decompression speeds, they quote "We know LZ4 is significantly faster than ZSTD on standalone benchmarks: likely bottleneck is ROOT IO API"

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#62

Post by forgotpwd16 »

>better, faster

If I'm reading the benchmarks correctly, it gets higher compression but is slower and has higher memory usage. Thus cannot call it better.

>spiritual successor to BZip2

What does that mean? If it isn't related to bzip2, why choose this name?

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#63

Post by forgotpwd16 »

yakubin wrote: post #60

From the "disclaimers" section: > Every compression of a file implies an assumption that the compressed file can be decompressed to reproduce the original. Great efforts in design, coding and testing have been made to ensure that this program works correctly. > However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probab…

Cannot answer your question but, since you mentioned it, there's a mathematical specification of deflate (see: https://arxiv.org/abs/1609.01220).

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#64

Post by Matumio »

sounds wrote: post #57
hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

Can you test it out and post results back here?

No, but it would be nice to see visually where it is on the size vs (de-)compression speed pareto-front. Like this graphic (from the zstd homepage): https://raw.githubusercontent.com/facebook/zstd/master/doc/i...

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#66

Post by [deleted] »

hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

[deleted]

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#67

Post by usefulcat »

klauspost wrote: post #33

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…

If you’re using xz, pixz can do multithreaded decompression. It’s still xz/lzma, so still expensive to decompress, but at least that allows you to throw as many cores as you want at it.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#68

Post by linsomniac »

sounds wrote: post #57
hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

Can you test it out and post results back here?

For linux-5.17.6.tar:

Original file: 129MB xz, 1.2G uncompressed.

"zstd -T0": 1.34 seconds, 189M

"xz -T0": 63 seconds, 131M

"xz -T0 -9": 183 seconds, 125M

"bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV)

"bzip3 -e": 84 seconds, 129M

I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#69

Post by adrianmonk »

bayindirh wrote: post #44
klauspost wrote: post #33

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…

> I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this. I think it boils down to the feelings of the author (of the previous format). I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Similarly if someone created a follow-up file format to something I've designed, I'd just…

> I don't think PKWARE feels bad because ZSTD is a homage to ZIP.

Is zstd actually an homage to zip?

I'm not saying that it definitely isn't, but the only connection I know of myself is that they both begin with the letter Z, and the letter Z has a long association with data compression that goes back before the zip format / pkzip program.

The LZ77, LZ78[1], and LZW[2] algorithms all predate the zip format. As do two very old, obsolete Unix compression programs: "pack"[3], which is uses a ".z" suffix for compressed files, and "compress"[4], which uses a ".Z" suffix for compressed files.

In those algorithm names, the L and Z stand for Lempel and Ziv, respectively. But interestingly, the Unix "pack" program uses a ".z" suffix even though its algorithm is just Huffman (not one of the Lempel-Ziv family of algorithms), so the letter Z somehow came to signify data compression more generally.

Rough timeline of letter Z in data compression:

1977: LZ77

1978: LZ78

1982 or earlier: Unix "pack" (.z)

1984: LZW

1985: Unix "compress" (.Z)

1989: PKZIP

---

[1] https://en.wikipedia.org/wiki/LZ77_and_LZ78

[2] https://en.wikipedia.org/wiki/Lempel-Ziv-Welch

[3] https://en.wikipedia.org/wiki/Pack_(compression)

[4] https://en.wikipedia.org/wiki/Compress

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#70

Post by palaiologos »

linsomniac wrote: post #68
sounds wrote: post #57

Earlier quoted context omitted.

Can you test it out and post results back here?

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

it's `bzip3 -e -j 6`. you need a space.
Post reply on HN