Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

31–40 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#31

Post by cout »

wongarsu wrote: post #21
joelthelion wrote: post #7

In the Era of zstandard, do we really need this?

I find it somewhat telling that they don't benchmark themselves against zstd. Right now I'm almost exclusively using zstd (general stuff) or lzma2/xz (high compression where read speed doesn't matter). And of course gz and zip for data interchange where compatibility is key. From the information presented bzip3 won't replace any of those use cases for me, but that's fine. Maybe it fits somebody else's use case, or ma…

Have you ever tried lzma/lzma2 with the hc3 (hash chain) match finder instead of the default (bt3 or bt4) match finder? I've found this to be a really good middle ground between gz/deflate and lzma2 with default settings.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#32

Post by joppy »

palaiologos wrote: post #30
joppy wrote: post #29

Why is there such a big disclaimer/warning on the front? Shouldn’t the program just check that decompress(compress(x)) = x as it goes, and then it can be sure that compress(x) has not lost any data?

no compressor tests the output while compressing as it hurts the performance. you can do it after compressing, though, using `bzip3 -t`.

Right, but I would probably test the output while compressing instead of putting a big USE AT YOUR OWN PERIL sign across the front…

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#33

Post by klauspost »

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed.

Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data.

I am not sure how I feel about you "stealing" the bzip name. While the author of bzip2 doesn't seem to plan to release a follow-up, I feel it is bad manner to take over a name like this.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#34

Post by klauspost »

thilog wrote: post #20
once_inc wrote: post #10

Earlier quoted context omitted.

That's what I took out of it too. Sacrificed a bit of speed and a lot of memory for a smaller output size. edit: ah, bzip3 is parallelizable, while bzip2 isn't. That alone is enough for me to be able to claim 'faster'.

bzip2 can exploit concurrency through pbzip2, can't it?

It is kind of a hack for decompression, where you look forward in the stream for a block signature, and try decompressing from there.

In practice it works, but it isn't pretty ;)

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#35

Post by vintermann »

klauspost wrote: post #33

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…

With the other parts of the codec, I doubt it's possible for files compressed with this, but one of the strengths of BWT-based compression is that there has been a lot of research on search operations directly on compressed data.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#38

Post by 149764 »

palaiologos wrote: post #22
wongarsu wrote: post #21

Earlier quoted context omitted.

I find it somewhat telling that they don't benchmark themselves against zstd. Right now I'm almost exclusively using zstd (general stuff) or lzma2/xz (high compression where read speed doesn't matter). And of course gz and zip for data interchange where compatibility is key. From the information presented bzip3 won't replace any of those use cases for me, but that's fine. Maybe it fits somebody else's use case, or ma…

zstd -19 linux.tar 462.58s user 0.76s system 100% cpu 217M memory 7:42.56 total % wc -c linux.tar.zst linux.bz3 134980904 linux.tar.zst 129255792 linux.bz3

  # compression

  bzip3 -j 4 -e linux-5.18-rc6.tar linux-5.18-rc6.tar.bz3 
    user: 345.48s system: 0.59s cpu: 373% total: 1:32.75

  zstd -19 --long -T4 -f linux-5.18-rc6.tar
    user: 1270.48s system: 0.89s cpu: 376% total: 5:37.9

  > du -b linux-5.18-rc6.tar.* | sort -rn | reln
  1.000000  130907738  linux-5.18-rc6.tar.zst
  0.994715  130215881  linux-5.18-rc6.tar.bz3
With additional ‘--ultra -22’ tar.zst is smaller, but the compression time sky rockets.

  # decompression 

  bzip3 -j 4 -d linux-5.18-rc6.tar.bz3 linux-5.18-rc6.tar 
    user: 222.57s system: 0.92s cpu: 362% total: 1:01.69

  bzip3 -d linux-5.18-rc6.tar.bz3 linux-5.18-rc6.tar 
    user: 141.29s system: 0.89s cpu: 99% total: 2:22.19

  zstd -d -T4 -f linux-5.18-rc6.tar.zst 
    user: 2.26s system: 0.84s cpu: 99% total: 3.102
zstd doesn’t seem to support parallel decoding, but still 20x faster

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#39

Post by trasz »

palaiologos wrote: post #18
trasz wrote: post #17

Earlier quoted context omitted.

Not to mention the restrictive license which effectively prohibits its use in any Open Source project licensed under anything other than GPLv3.

Frankly, same holds for gzip. I've been planning to relicense bzip3 with the more permissive LGPLv3.

Gzip is just a frontend for zlib, which is BSD(ish).

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#40

Post by perihelions »

klauspost wrote: post #33

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…

That's funny, 23 MiB/s is exactly what I get for reading systemd logs (on an NVME SSD). Is it supposed to be otherwise?

    $ sudo journalctl -r | pv -a > /dev/null
    [22.8MiB/s]
Post reply on HN