Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3: A spiritual successor to BZip2

github.com

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Bzip3: A spiritual successor to BZip2

#91

Post by linsomniac »

palaiologos wrote: post #82
linsomniac wrote: post #19

Earlier quoted context omitted.

My standard test is compressing a "dd" disc image of a Linux install (I use these for work), with unused blocks being zeroed. Results: Uncompressed: 7,516,192,768 zstd: 1,100,323,366 bzip3 -b 511 -j 4: 1,115,125,019

Hi, tool author here! Thank you for your benchmark! As you may be aware, different compression tools fill in different data type niches. In particular, less specialised statistical methods (bzip2, bzip3, PPMd) generally perform poorly on vaguely defined binary data due to unnatural distribution of the underlying data that at least in bzip3's case does not lend well to suffix sorting. Conversely, Lempel-Ziv methods us…

Thanks for the reply. I just figured I'd try it and see, and the bzip3 results are extremely good. I figured it was worth trying because a fair bit of the data in that image is non-binary (man pages, config files, shell/python code), but probably the bulk of it is binary (kernel images, executables).

Re: Bzip3: A spiritual successor to BZip2

#92

Post by otterley »

froh wrote: post #40
otterley wrote: post #3

(2022)

what do you mean?? usually this signifies the year some article was finalized and published. and this is a GitHub repo with recent commits so this is not correct here.

The date signifies that this tool is not new; and this is ostensibly a "news" site.

Re: Bzip3: A spiritual successor to BZip2

#93

Post by Brian_K_White »

adastra22 wrote: post #74
hinkley wrote: post #73

Earlier quoted context omitted.

How far back?

How much does that matter? All currently supported releases at least.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago.

It's a fair question, and "currently supported" is essentially meaningless.

(Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are both true and both beside the point. Fact is still, this is not only a "currently supported" system by some technicality like the final months of a 5 year lts release or something, this is the up to this very minute version of a massive install base, with a 20 year old version of something as basic and all-touching as the very shell itself.

I know about this so intimately because I actually have scripts that don't work because they were intentional stunts to see just what could be done without forking any external exes or even child shells, but allow using every possible feature and trick in bash itself.)

Re: Bzip3: A spiritual successor to BZip2

#94

Post by hinkley »

Brian_K_White wrote: post #93
adastra22 wrote: post #74

Earlier quoted context omitted.

How much does that matter? All currently supported releases at least.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago. It's a fair question, and "currently supported" is essentially meaningless. (Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are…

I really don’t want to switch to zsh, but these are facts.

The number of devs working on OSX producing non-OSX applications is staggering.

Re: Bzip3: A spiritual successor to BZip2

#95

Post by adastra22 »

Brian_K_White wrote: post #93
adastra22 wrote: post #74

Earlier quoted context omitted.

How much does that matter? All currently supported releases at least.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago. It's a fair question, and "currently supported" is essentially meaningless. (Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are…

What does that have to do with zstd?

Re: Bzip3: A spiritual successor to BZip2

#96

Post by vanviegen »

loxias wrote: post #68
mmastrac wrote: post #4

I've studied the Burrows-Wheeler Transform, I understand the transformation, I've re-implemented it countless times for kicks, I see how it improves compressability, but for the life of me the intuition of _why_ it works has never really clicked. It's a fantastic bit of algorithmic magic that will always impress me to see it.

I always understood it as working because of the predictability of a symbol/letter/token given the previous one. Sorting all the shifts of a string puts all the characters in order, then looking at the last column shows you all the _preceding_ characters. If there's any predictability there (which there often is), it's now easier to compress. It's sorta like an entropy coder in that way. I've never thought of it as b…

Understanding why increasing predictability helps with compression is not the hard part though. What's hard to grasp is why the transform is reversible.

Re: Bzip3: A spiritual successor to BZip2

#97

Post by loeg »

Brian_K_White wrote: post #93
adastra22 wrote: post #74

Earlier quoted context omitted.

How much does that matter? All currently supported releases at least.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago. It's a fair question, and "currently supported" is essentially meaningless. (Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are…

The bash 4 thing is due to the GPL 3, not some inherent slowness in updating software. It has nothing to do with zstd, which is permissively licensed.

Re: Bzip3: A spiritual successor to BZip2

#98

Post by Brian_K_White »

loeg wrote: post #97
Brian_K_White wrote: post #93

Earlier quoted context omitted.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago. It's a fair question, and "currently supported" is essentially meaningless. (Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are…

The bash 4 thing is due to the GPL 3, not some inherent slowness in updating software. It has nothing to do with zstd, which is permissively licensed.

It doesn't matter why. All that matters is that "current" is not a valid word. Old things exist in "current" systems. And current systems may also be old systems.

Re: Bzip3: A spiritual successor to BZip2

#99

Post by Brian_K_White »

adastra22 wrote: post #95
Brian_K_White wrote: post #93

Earlier quoted context omitted.

bash4 was released Feb 2009. 16 years ago. The very latest version of osx ships with bash3, from 20 years ago. It's a fair question, and "currently supported" is essentially meaningless. (Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are…

What does that have to do with zstd?

It directly addresses "How much does that matter? All currently supported releases at least."

Re: Bzip3: A spiritual successor to BZip2

#100

Post by eqvinox »

linsomniac wrote: post #2

Highlight (benchmark of Perl source code): The results follow: xz -T16 -9 -k - 2'056'645'240 bytes (c=12m09s, d=4m40s) bzip2 -9 -k - 3'441'163'911 bytes (c=17m16s, d=9m22s) bzip3 -b 256 - 1'001'957'587 bytes (c=7m10s, d=4m6s? Unclear on source page) bzip3 -b 511 - 546'456'978 bytes (c=7m08s, d=4m6s? Unclear) zstd -T12 -16 - 3'076'143'660 bytes (c=6m32s, d=3m51s) edit: Adding times and compression levels

I'd argue that's actually the lowlight of the README since that is a very poor choice of benchmark. Combining a multitude of versions of the same software massively favors an algorithm good at dealing with this kind of repetitiveness in a way that will not be seen in typical applications.

The "Corpus benchmarks" further down in the README are IMHO much more practically relevant. The compression ratio of bzip3 is not significantly better, but the runtime seems quite a bit lower than lzma at least.

Post reply on HN