Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3 – A better and stronger spiritual successor to bzip2

github.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#91

Post by adrianmonk »

bayindirh wrote: post #73
adrianmonk wrote: post #69

Earlier quoted context omitted.

> I don't think PKWARE feels bad because ZSTD is a homage to ZIP. Is zstd actually an homage to zip? I'm not saying that it definitely isn't, but the only connection I know of myself is that they both begin with the letter Z, and the letter Z has a long association with data compression that goes back before the zip format / pkzip program. The LZ77, LZ78[1], and LZW[2] algorithms all predate the zip format. As do two…

> Is zstd actually an homage to zip? I think so. Or more generally, DEFLATE family of algorithms used by zlib and ZIP. > I'm not saying that it definitely isn't, but the only connection I know of myself is that they both begin with the letter Z, and the letter Z has a long association with data compression that goes back before the zip format / pkzip program. > The LZ77, LZ78[1], and LZW[2] algorithms all predate the…

Right, there's definitely a connection in the algorithms. They're all derivatives of the same family of stuff.

I could've been clearer, but when I said the only connection I saw between them was the Z, I meant the only connection between their names. In other words, while zstd surely has zip among its main influences, I'm not sure that the Z in "zstd" is there to convey "zip-like". It could be, or maybe it's just there because Z is generally associated with data compression.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#93

Post by joveian »

bell-cot wrote: post #59
mschuster91 wrote: post #14

Earlier quoted context omitted.

One should verify the integrity of stuff like backups or archives anyway, by supplying the end user with a sha1 or better hash of both the compressed/encrypted archive as well as all of the files it contains, and by regularly verifying if both still match.

Yes...though I'd say to rule out sha1, or any other "no longer considered secure" hashes. The space & time savings (vs, say, sha-512) are really not worth baking into your backup format & procedures. Keep in mind that you might need to really verify the integrity of your backup during a ransomware incident, or as part of a high-stakes legal situation, or ...

SHA1 and even MD5 are broken for collision resistance, not second preimage (which is what matters for backups). While it can still make sense to not use SHA1 for anything, it is fine if you do use it. Blake3 is the one to use if you are looking for maxium speed, although it is newer and some may avoid it for that. In a quick test (using hyperfine -w 1 -m 3) on a 8GB file (Arch Linux, i5-6260U 2 core processor with hyperthreading disabled) times are 1.995s for b3sum, 11.847s for sha1sum, 14.495s for b2sum, 16.903s for sha512sum, and 24.737s for sha256sum. md5sum for comparison is 16.492s, just a tiny bit slower than sha512sum, so that is why you rarely see it these days even for things like backups. SHA3-224 (sha3sum) took 39.531s and SHA3-512 (sha3sum -a 512) took 73.164s, although they will eventually be very quick with hardware support.

xz -C sha256 will use sha256 on the uncompressed data.

Personally, I use mtree inside the backup file (from pkgsrc bootstrap on Linux), although it has trouble with a few unicode file names. That kind of tool is great and I'm not sure why there doesn't seem to be an equivalent in the Linux world.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#95

Post by eru »

bayindirh wrote: post #75
eru wrote: post #50

Earlier quoted context omitted.

> Open source software is designed to be mangled, modified, shared and leapfrogged. I agree in spirit, but I can also see why someone might want their source to be free and mangleable, but still care about trademarks. (Just imagine Linus Torvalds getting lots of emails with support requests for a hypothetical Linux2 operating system that I wrote, and that he has no relation with. That could become pretty annoying; ev…

A trademark changes the situation in every aspect. If your code is getting big, famous, and needs its name and likeness (e.g. Firefox), you get a trademark, and your derivatives shall not use the name. It's simple and clear. When I quickly looked, bzip2 didn't have a trademark, and I assumed the developer don't care. It's same for me. If I care, I'd trademark it, and prevent people from using it. If I'm giving the na…

So, a trademark is a 'legal' concept, and comes with certain rights and obligations. Eg you need to actively defend your trademark in order to keep it.

I was perhaps a bit fuzzy: I used the word trademark, but I meant the moral equivalent that only confers moral rights and obligations. Ie don't be a jerk and name your stuff in a confusing way, even if there's no legal obligation.

Even this fuzzier 'moral' trademark only applies, if the original owner wants it to apply. I have no special insight into the bzip2 situation. So I have no clue whether the authors of bzip or bzip2 cared. I gladly defer to your research on this question.

My point was meant purely abstract about the possibility of wanting your source to be free and open and fondled by others, but still have a (morally) protected name.

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#96

Post by rincebrain »

loeg wrote: post #90
klodolph wrote: post #58

Earlier quoted context omitted.

We must be looking at different graphs. See p.7 https://indico.fnal.gov/event/16264/contributions/36466/atta... You can see the classic Pareto frontier, with LZ4 filling the niche at the very bottom right edge of the graph.

I'm looking at http://facebook.github.io/zstd/ "Benchmarks." P.7 of your slides doesn't seem to cover Zstd --fast or decompression. It would be interesting to see how Zstd performs in those modes.

Depending on your data, experimentally, some people find zstd --fast can beat LZ4 for them on compression, some people find the opposite; my usual advice to people considering one or the other is to experiment and find out.

(An interesting anecdote about the differing notions of compressibility - when I recently wrote something to do a clever dance to avoid burning a great deal of CPU on incompressible data with higher zstd levels, I ended up using LZ4 -> zstd-1 as a two-tier filter to catch incompressible data, because what they each thought was incompressible was different enough that only using LZ4 lost a significant amount of compression sometimes, but only using zstd-1 was comparatively expensive and also lost a significant fraction.)

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#97

Post by Klasiaster »

gjvc wrote: post #45
thilog wrote: post #20

Earlier quoted context omitted.

bzip2 can exploit concurrency through pbzip2, can't it?

see also: lbzip2 (context: I have had a situation where files created by pbzip2 on linux were not able to be decompressed with some library on .NET, but using lbzip2, they were. I never looked into the details.)

Can't agree more, lbzip2 is the go-to tool for dealing with bzip2 compression and decompression, it's a whole lot of faster than bzip2 which is single-threaded!

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#99

Post by linsomniac »

palaiologos wrote: post #74
linsomniac wrote: post #71

Earlier quoted context omitted.

One more note: "bzip3 -e -j 6 -b 50": 25 seconds, 125MB So nearly as good as the best of xz, but in a 20th the time. However: Do note that any unexpected use is met with a SIGSEGV: using as a filter, using "-j6" instead of "-j 6", not specifying "-e"...

lies. not specifying -e displays an error message: % bzip3 -e -j 6 -b 50 corpus/calgary.tar % bzip3 -j 6 -b 50 corpus/calgary.tar bzip3 - A better and stronger spiritual successor to bzip2. Copyright (C) by Kamila Szewczyk, 2022. Licensed under the terms of GPLv3. Usage: bzip3 [-e/-d/-t/-c] [-b block_size] input output Operations: -e: encode -d: decode -t: test Extra flags: -c: force reading/writing from standard str…

You're right, what I thought was encoding/filter SEGVs were the bug in handling "-j6".

Re: Bzip3 – A better and stronger spiritual successor to bzip2

#100

Post by linsomniac »

adgjlsfhk1 wrote: post #78
linsomniac wrote: post #68

Earlier quoted context omitted.

For linux-5.17.6.tar: Original file: 129MB xz, 1.2G uncompressed. "zstd -T0": 1.34 seconds, 189M "xz -T0": 63 seconds, 131M "xz -T0 -9": 183 seconds, 125M "bzip3 -e -j 6": 21 seconds, 129M (edited, was SIGSEGV) "bzip3 -e": 84 seconds, 129M I used linux source because the source website uses linux and recommends bzip3 for compressing source and text. Results were on Ubuntu 22.04, Intel(R) Core(TM) i7-10750H CPU @ 2.60…

Can you also give decompression speeds?

"zstd -d": 1.03 seconds

"xz -d": 7.92 seconds

"bzip3 -d" as filter: SEGSEGV

"bzip3 -d linux-5.17.6.tar.bz3": 81 seconds

"bzip -d -j 6": 21.35 seconds (edit)

Lest I be called a liar again:

  $ ./bzip3-1.0.1/bzip3 -d /dev/null
  fish: Job 1, 'time ./bzip3-1.0.1/bzip3 -d
Post reply on HN