Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#101

Post by palaiologos »

linsomniac wrote: post #100
adgjlsfhk1 wrote: post #78

Earlier quoted context omitted.

Can you also give decompression speeds?

"zstd -d": 1.03 seconds "xz -d": 7.92 seconds "bzip3 -d" as filter: SEGSEGV "bzip3 -d linux-5.17.6.tar.bz3": 81 seconds "bzip -d -j 6": 21.35 seconds (edit) Lest I be called a liar again: $ ./bzip3-1.0.1/bzip3 -d /dev/null fish: Job 1, 'time ./bzip3-1.0.1/bzip3 -d

Consider using `-c`, which makes the compressor use standard streams, or pull the main branch because I had just pushed a tiny patch that automatically enables it when no positional arguments are given.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#102

Post by jkbonfield »

It doesn't compare itself against bsc, which feels a bit poor IMO given it's using Grebnov's libsais and LZP algorithm (he's the author of libbsc).

On my own benchmarks, it's basically comparable size (about 0.1% smaller than bsc), comparable encode speeds, and about half the decode speed. Plus bsc has better multi-threading capability when dealing with large blocks.

Also see https://quixdb.github.io/squash-benchmark/unstable/ (and without /unstable for more system types) for various charts. No bzip3 there yet though.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#103

Post by palaiologos »

jkbonfield wrote: post #102

It doesn't compare itself against bsc, which feels a bit poor IMO given it's using Grebnov's libsais and LZP algorithm (he's the author of libbsc). On my own benchmarks, it's basically comparable size (about 0.1% smaller than bsc), comparable encode speeds, and about half the decode speed. Plus bsc has better multi-threading capability when dealing with large blocks. Also see https://quixdb.github.io/squash-benchmark…

You've literally tested it on a single file, enwik8. That's not enough to extrapolate valuable results. One of the benchmarks:

  time ./bsc e ../linux.tar linux.bsc -e2 -b16 -T
  68.69s user 1.14s system 99% cpu 117M memory 1:09.84 total
While bzip3 uses 98M, takes 1min 17s to produce a 129023171 byte file, compared to 127747834B from BSC. They're very similar except bzip3 tends to use less memory and decompresses a little slower. BSC is much more mature than bzip3 though, and the benchmarks might be a subject to change some time in the future. Surprisingly, BSC code isn't really that robust (I reported a UB bug to libsais and had to pretty much rework the LZP code because it couldn't stand fuzzing).

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#104

Post by alerque »

the-alchemist wrote: post #82

I'm really interested in GPU-based compression / decompression. Anyone know what the current SOTA GPU-based algorithms are, and why they haven't taken off? Brotli has gotten browser support, so it seems to my naive self that a GPU-based algorithm is just waiting take over.

GPU's are good at massively parallel tasks. Compression is, almost by definition, not a parallel problem. If you want speed you can break it up into chunks and if you are optimizing from throughput there are gains to be made there. But if you are optimizing for compression, the more chunks you break the task up into the less opportunity you have to find ways to compress it. For example a fast compression tool creating an archive of files might split up each file into a different thread which gets the job done fast, but it will loose out on huge gains in compression if there are common parts to files that could have been compressed if there were processed as a single blob. GPUs are designed to do lots of small chunks of work in parallel, CPUs are better at doing bigger jobs faster.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#105

Post by alerque »

forgotpwd16 wrote: post #62

>better, faster If I'm reading the benchmarks correctly, it gets higher compression but is slower and has higher memory usage. Thus cannot call it better. >spiritual successor to BZip2 What does that mean? If it isn't related to bzip2, why choose this name?

It is related to bzip2 in the sense of using the Burrows-Wheeler algorithm.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#106

Post by jkbonfield »

palaiologos wrote: post #103
jkbonfield wrote: post #102

It doesn't compare itself against bsc, which feels a bit poor IMO given it's using Grebnov's libsais and LZP algorithm (he's the author of libbsc). On my own benchmarks, it's basically comparable size (about 0.1% smaller than bsc), comparable encode speeds, and about half the decode speed. Plus bsc has better multi-threading capability when dealing with large blocks. Also see https://quixdb.github.io/squash-benchmark…

You've literally tested it on a single file, enwik8. That's not enough to extrapolate valuable results. One of the benchmarks: time ./bsc e ../linux.tar linux.bsc -e2 -b16 -T 68.69s user 1.14s system 99% cpu 117M memory 1:09.84 total While bzip3 uses 98M, takes 1min 17s to produce a 129023171 byte file, compared to 127747834B from BSC. They're very similar except bzip3 tends to use less memory and decompresses a litt…

Well yes it was one file, but it was stated as being good on text and enwik8 is a pretty standard test corpus for text compressors.

I could have done more, but it somewhat vindicated what I was saying really. It has a very similar core to bsc (based on the same code) and gives very similar file sizes as expected. Note you may wish to use bsc -tT to disable both forms of threading. I don't know if that changes memory usage any.

Have you tried making PRs back to libbsc github to fit the UB and fuzzing issues? I'm sure the author would welcome fixes given you've already done the leg work.

Anyway, please do consider benchmarking against libbsc. It's conspicuously absent given the shared ancestry.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#107

Post by palaiologos »

jkbonfield wrote: post #106
palaiologos wrote: post #103

Earlier quoted context omitted.

You've literally tested it on a single file, enwik8. That's not enough to extrapolate valuable results. One of the benchmarks: time ./bsc e ../linux.tar linux.bsc -e2 -b16 -T 68.69s user 1.14s system 99% cpu 117M memory 1:09.84 total While bzip3 uses 98M, takes 1min 17s to produce a 129023171 byte file, compared to 127747834B from BSC. They're very similar except bzip3 tends to use less memory and decompresses a litt…

Well yes it was one file, but it was stated as being good on text and enwik8 is a pretty standard test corpus for text compressors. I could have done more, but it somewhat vindicated what I was saying really. It has a very similar core to bsc (based on the same code) and gives very similar file sizes as expected. Note you may wish to use bsc -tT to disable both forms of threading. I don't know if that changes memory…

I haven't figured a libsais fix and my LZP fix changes the functionality a little (removes chunking for better compression at a rather small runtime cost), so I don't think the author would like me to submit it. I have opened tickets, though.
Post reply on HN