Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#51

Post by jeffbee »

iruoy wrote: post #36

So bzip2 and bzip3 focus on compressed size, lz4 on compression speed and zstd on decompression speed?

I don't know if that's really accurate. LZ4 is often faster on both sides, while usually having a larger compressed size. On most of the inputs listed at this benchmark, LZ4 is twice as fast for compression, 50% faster for decompression, while having a compressed size about 125% as large as Zstd. My rule of thumb is that zstd is good if you're going to store or transmit the result, while lz4 is the better choice if you're planning to compress and decompress exactly once without storing (i.e. as a transfer encoding between two network peers).

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#52

Post by erk__ »

perihelions wrote: post #40
klauspost wrote: post #33

Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow when you need to grep through gigabytes of data. Same for data analysis, you don't want your input speed to be this limited when analysing gigabytes of data. I am not sure how I feel abo…

That's funny, 23 MiB/s is exactly what I get for reading systemd logs (on an NVME SSD). Is it supposed to be otherwise? $ sudo journalctl -r | pv -a > /dev/null [22.8MiB/s]

I think the lack of speed here is more that it has to serialize the data from disk into a readable format. I assume using the `--grep=` option is faster than piping it through grep because of this

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#53

Post by loeg »

klodolph wrote: post #24
lynguist wrote: post #19

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy. [1] https://github.com/lz4/lz4

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

My impression was that lz4 ratios were still marginally better than zstd for the same compression speed, and decompression is much, much faster.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#54

Post by klauspost »

yakubin wrote: post #46
perihelions wrote: post #40

Earlier quoted context omitted.

That's funny, 23 MiB/s is exactly what I get for reading systemd logs (on an NVME SSD). Is it supposed to be otherwise? $ sudo journalctl -r | pv -a > /dev/null [22.8MiB/s]

That appears to be systemd being slow. $ dd if=/dev/urandom of=test bs=1G count=1 iflag=fullblock $ gzip -k test $ zcat test.gz | pv -a >/dev/null [ 228MiB/s] $ sudo journalctl -r | pv -a >/dev/null [13.1MiB/s] UPDATE: Gzip with more real-world data[1]: $ gzip -k adventures-of-huckleberry-finn.txt $ zcat adventures-of-huckleberry-finn.txt.gz | pv -a >/dev/null [ 151MiB/s] [1]: https://gutenberg.org/files/76/76-0.txt…

By "compressing" random data you are bypassing gzip, since it will just store your data as uncompressed blocks, making "decompression" a memcopy.

With real data, deflate maxes out somewhere around there either way, but that is a bit coincidental.

With modern CPUs getting increasingly smaller IPC improvements this will likely be pretty much the max decompression speed we can expect from gzip going forward.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#55

Post by yakubin »

klauspost wrote: post #54
yakubin wrote: post #46

Earlier quoted context omitted.

That appears to be systemd being slow. $ dd if=/dev/urandom of=test bs=1G count=1 iflag=fullblock $ gzip -k test $ zcat test.gz | pv -a >/dev/null [ 228MiB/s] $ sudo journalctl -r | pv -a >/dev/null [13.1MiB/s] UPDATE: Gzip with more real-world data[1]: $ gzip -k adventures-of-huckleberry-finn.txt $ zcat adventures-of-huckleberry-finn.txt.gz | pv -a >/dev/null [ 151MiB/s] [1]: https://gutenberg.org/files/76/76-0.txt…

By "compressing" random data you are bypassing gzip, since it will just store your data as uncompressed blocks, making "decompression" a memcopy. With real data, deflate maxes out somewhere around there either way, but that is a bit coincidental. With modern CPUs getting increasingly smaller IPC improvements this will likely be pretty much the max decompression speed we can expect from gzip going forward.

It actually made the file bigger (1.1G from 1.0G) :)

I was getting the same numbers with text data I have scattered on my disk, but those were small, so I decided to generate a bigger file. But, yes, I agree a more robust benchmark would use a Mark Twain novel e.g.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#56

Post by perihelions »

erk__ wrote: post #52
perihelions wrote: post #40

Earlier quoted context omitted.

That's funny, 23 MiB/s is exactly what I get for reading systemd logs (on an NVME SSD). Is it supposed to be otherwise? $ sudo journalctl -r | pv -a > /dev/null [22.8MiB/s]

I think the lack of speed here is more that it has to serialize the data from disk into a readable format. I assume using the `--grep=` option is faster than piping it through grep because of this

    --grep=
That's exactly what I needed to know! I'm glad I asked the stupid question. Thank you!

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#57

Post by sounds »

hannob wrote: post #48

It seems somewhat suspicious that the benchmarks don't compare to zstd. It's not entirely clear to me what the selling point is. "Better than bzip2" isn't exactly a convincing sales pitch given bzip2 is mostly of historic interest these days. Right now the modern compression field is basically covered by xz (if you mostly care about best compression ratio) and zstd (if you want decent compression and very good speed)…

Can you test it out and post results back here?

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#58

Post by klodolph »

loeg wrote: post #53
klodolph wrote: post #24

Earlier quoted context omitted.

Zstandard achieves similar speeds at higher ratios. LZ4 only comes out ahead if you use LZ4 at, like, level 1.

My impression was that lz4 ratios were still marginally better than zstd for the same compression speed, and decompression is much, much faster.

We must be looking at different graphs. See p.7

https://indico.fnal.gov/event/16264/contributions/36466/atta...

You can see the classic Pareto frontier, with LZ4 filling the niche at the very bottom right edge of the graph.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#59

Post by bell-cot »

mschuster91 wrote: post #14
kstenerud wrote: post #12

There comes a point where the complexity itself becomes too much of a liability. It's important to be able to trust these algorithms as well as all popular implementations with your data.

One should verify the integrity of stuff like backups or archives anyway, by supplying the end user with a sha1 or better hash of both the compressed/encrypted archive as well as all of the files it contains, and by regularly verifying if both still match.

Yes...though I'd say to rule out sha1, or any other "no longer considered secure" hashes. The space & time savings (vs, say, sha-512) are really not worth baking into your backup format & procedures. Keep in mind that you might need to really verify the integrity of your backup during a ransomware incident, or as part of a high-stakes legal situation, or ...

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#60

Post by yakubin »

From the "disclaimers" section:

> Every compression of a file implies an assumption that the compressed file can be decompressed to reproduce the original. Great efforts in design, coding and testing have been made to ensure that this program works correctly.

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.

That got me thinking: I've always implicitly assumed that authors of lossless compression algorithms write mathematical proofs that D o C = id[1]. However, now that I've started looking, I can't seem to find that even for Deflate. What is the norm?

[1]: C being the compression function, D being the decompression function, and o being function composition.

Post reply on HN