Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

bzip3

github.com

Post reply on HN
Print view View original

21–30 of 131 posts

Re: bzip3

#21

Post by ot »

CJefferson wrote: post #19
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

> 512GB is a huge block size for bzip3

Sorry! That was a typo, it should have been 512MB (now fixed). Still huge.

Re: bzip3

#22

Post by vintermann »

adrian_b wrote: post #8
sylware wrote: post #3

Isn't that XZ?

No. It is an unrelated algorithm.

It's a very strange algorithm completely different from most compression methods, that's what makes it interesting IMO. But it's probably not realistically competitive, since one needs to do more after the burrows-wheeler transform, and all that "more" has been ridiculously more optimized in zstd and other modern compression methods. Compressing bwt-transformed data is easier, but that doesn't mean it's easy to further than what's easy.

Re: bzip3

#23

Post by [deleted] »

CJefferson wrote: post #19
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

[deleted]

Re: bzip3

#26

Post by 8organicbits »

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.

I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...

Re: bzip3

#27

Post by optionalsquid »

ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

How does memory usage compare between your two runs?

The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising

Re: bzip3

#29

Post by nvme0n1p1 »

ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

> 8MB I believe for high levels

Yep, i found it in the source here:

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

Also, zstd docs say:

> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

Re: bzip3

#30

Post by nvme0n1p1 »

8organicbits wrote: post #26

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.

I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

Post reply on HN