Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

21–30 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#21

Post by wongarsu »

joelthelion wrote: post #7

In the Era of zstandard, do we really need this?

I find it somewhat telling that they don't benchmark themselves against zstd.

Right now I'm almost exclusively using zstd (general stuff) or lzma2/xz (high compression where read speed doesn't matter). And of course gz and zip for data interchange where compatibility is key. From the information presented bzip3 won't replace any of those use cases for me, but that's fine. Maybe it fits somebody else's use case, or maybe it's the foundation for the next great algorithm that we all end up using.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#22

Post by palaiologos »

wongarsu wrote: post #21
joelthelion wrote: post #7

In the Era of zstandard, do we really need this?

I find it somewhat telling that they don't benchmark themselves against zstd. Right now I'm almost exclusively using zstd (general stuff) or lzma2/xz (high compression where read speed doesn't matter). And of course gz and zip for data interchange where compatibility is key. From the information presented bzip3 won't replace any of those use cases for me, but that's fine. Maybe it fits somebody else's use case, or ma…

zstd -19 linux.tar 462.58s user 0.76s system 100% cpu 217M memory 7:42.56 total

% wc -c linux.tar.zst linux.bz3 134980904 linux.tar.zst 129255792 linux.bz3

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#23

Post by chungy »

palaiologos wrote: post #18
trasz wrote: post #17

Earlier quoted context omitted.

Not to mention the restrictive license which effectively prohibits its use in any Open Source project licensed under anything other than GPLv3.

Frankly, same holds for gzip. I've been planning to relicense bzip3 with the more permissive LGPLv3.

gzip has BSD-licensed compatible alternatives already. It's doubtful the same attention would be given to bzip3; chicken-and-egg scenario there. Plus the lingering question of "Why not zstd?"

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#24

Post by klodolph »

lynguist wrote: post #19

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy. [1] https://github.com/lz4/lz4

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#25

Post by Beltalowda »

klodolph wrote: post #24
lynguist wrote: post #19

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy. [1] https://github.com/lz4/lz4

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

It's still quite a bit faster than zstd -1, at least according to their GitHub page. The trade-off is it's a worse compression ratio (2.8 vs. 2.1), but in some cases that's a good trade-off.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#26

Post by adgjlsfhk1 »

klodolph wrote: post #24
lynguist wrote: post #19

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy. [1] https://github.com/lz4/lz4

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

zstd has similar compression speeds, but lz4 crushed everything else for decompression.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#27

Post by klauspost »

klodolph wrote: post #24
lynguist wrote: post #19

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy. [1] https://github.com/lz4/lz4

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

Yes, zstd has forced bitwise match coding, whereas lz4 is byte-aligned and with inline literals.

So lz4 has some base advantages in terms of speed, which zstd is unlikely to match. But as you point out it is only relevant for very high speed operations.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#28

Post by Beltalowda »

baybal2 wrote: post #16
joelthelion wrote: post #7

In the Era of zstandard, do we really need this?

1. zStandard is not a standard 2. Bzip2 is somewhat is a standard 3. zStandard is not a substitute for Bzip2

In what way is bzip2 more of a "standard" than zstd? bzip2 doesn't even seem to have any official reference description of its file format; just an "unofficial" one[1], whereas zstd is RFC 8478[2].

When I evaluated various compression algorithms a few years ago zstd came ahead of bzip2 in every metric.

[1]: https://github.com/dsnet/compress/blob/master/doc/bzip2-form...

[2]: https://datatracker.ietf.org/doc/html/rfc8478

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#30

Post by palaiologos »

joppy wrote: post #29

Why is there such a big disclaimer/warning on the front? Shouldn’t the program just check that decompress(compress(x)) = x as it goes, and then it can be sure that compress(x) has not lost any data?

no compressor tests the output while compressing as it hurts the performance. you can do it after compressing, though, using `bzip3 -t`.
Post reply on HN