In the Era of zstandard, do we really need this?
Yes, because someone said the same when zstandard came out. This may not have the same strong points, but maybe the next will… compression is not a completed task.
Bzip3 – A better and stronger spiritual successor to bzip2
11–20 of 107 posts
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#12There comes a point where the complexity itself becomes too much of a liability. It's important to be able to trust these algorithms as well as all popular implementations with your data.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#13Interesting, this seems to be a good replacement for xz if the benchmarks are representative.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#14There comes a point where the complexity itself becomes too much of a liability. It's important to be able to trust these algorithms as well as all popular implementations with your data.
One should verify the integrity of stuff like backups or archives anyway, by supplying the end user with a sha1 or better hash of both the compressed/encrypted archive as well as all of the files it contains, and by regularly verifying if both still match.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#15Earlier quoted context omitted.
That discussion doesn't really clear up my confusion though. I don't understand how bzip3 gets to claim "A better, faster and stronger spiritual successor to BZip2." when even all its own benchmarks show it's slower than bzip2?
bzip3 usually operates on bigger block sizes, up to 16 times bigger than bzip2. additionally, bzip3 supports parallel compression/decompression out of the box. for fairness, the benchmarks have been performed using single thread mode, but they aren't quite as fair towards bzip3 itself, as it uses a way bigger block size. what bzip3 aims to be is a replacement for bzip2 on modern hardware. what used to not be viable d…
First, I don't understand what the block size has to do with the number of threads used. I understand why one could consider the benchmarks unfair to bzip3 because they're single-threaded (depending on exactly one defines "faster"), but why do you say they are not fair towards bzip3 because of the bigger block size it uses? Do the benchmarks not use the optimal block size for each compression tool?
I can see how bzip3 is better able to exploit the characteristics of modern hardware, but that's not enough to call it faster. The proof of the pudding is in the eating, and if the benchmarks show bzip3 is slower than bzip2 than bzip3 is clearly not faster (but perhaps "aiming to be faster", with some work to be done before it reaches that goal).
Better compression at a slightly slower pace can indeed be a good trade off. I'm not saying bzip3 doesn't work well. What I'm saying is that the "faster" in its description "A better, faster and stronger spiritual successor to BZip2" is not supported by the evidence.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#16In the Era of zstandard, do we really need this?
1. zStandard is not a standard
2. Bzip2 is somewhat is a standard
3. zStandard is not a substitute for Bzip2
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#17In the Era of zstandard, do we really need this?
Not to mention the restrictive license which effectively prohibits its use in any Open Source project licensed under anything other than GPLv3.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#18In the Era of zstandard, do we really need this?
Not to mention the restrictive license which effectively prohibits its use in any Open Source project licensed under anything other than GPLv3.
Frankly, same holds for gzip. I've been planning to relicense bzip3 with the more permissive LGPLv3.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#19If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy.
Re: Bzip3 – A better and stronger spiritual successor to bzip2
#20From their own benchmarks it seems more like bzip3 is geared towards a different compression/speed trade-off than bzip2, rather than an unambiguous all-around improvement. Am I misreading it?
That's what I took out of it too. Sacrificed a bit of speed and a lot of memory for a smaller output size. edit: ah, bzip3 is parallelizable, while bzip2 isn't. That alone is enough for me to be able to claim 'faster'.
bzip2 can exploit concurrency through pbzip2, can't it?