Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

bzip3

github.com

Post reply on HN
Print view View original

61–70 of 130 posts

Re: bzip3

#61

Post by tredre3 »

benatkin wrote: post #50
esseph wrote: post #34

Earlier quoted context omitted.

”Not really" what? It's hard to understand what point you're trying to make. Can you clarify?

It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.

I think you're being a bit pedantic.

A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.

Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.

Re: bzip3

#62

Post by vlovich123 »

8organicbits wrote: post #60
nvme0n1p1 wrote: post #30

Earlier quoted context omitted.

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s

At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.

Re: bzip3

#63

Post by out_of_protocol »

8organicbits wrote: post #60
nvme0n1p1 wrote: post #30

Earlier quoted context omitted.

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s

zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different

Re: bzip3

#64

Post by sltkr »

ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?

I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.

Re: bzip3

#65

Post by sedatk »

finaard wrote: post #20
sedatk wrote: post #7

The latest release is a year ago, the last commit is two months ago, and the build is failing. The claim “stronger than bzip2” is strange. What does it even mean? Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.

Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page. pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP suppo…

> pbzip2 only can do parallel decompression on archives created with pbzip2

Oh I didn’t know that.

Re: bzip3

#66

Post by cesarb »

nvme0n1p1 wrote: post #29
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits so…

> They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

It's actually 8 bits: https://www.rfc-editor.org/rfc/rfc8878.html#name-window-desc...

These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).

Re: bzip3

#67

Post by ibejoeb »

jubilanti wrote: post #48
sergiotapia wrote: post #36

"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE." So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

Is that all that different than the standard MIT License: > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....

Practically, no. But that specific disclaimer could lead on to think that there may have been some observed data corruption in practice that isn't disclosed. I have no idea if there is, but I'm not keen to discover it myself.

Also practically, this isn't MIT. It is LGPL 3, which I believe includes the warranty terms of GPL 3, i.e., no warranty. So we're in the same place anyway.

Re: bzip3

#68

Post by loeg »

sltkr wrote: post #64
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)? I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.

If you want a 16MB window, use `--long=24` (2^24 is 16M). (I believe this is larger than the default window for zstd level 3, but smaller than the default window at higher compression levels.)

Re: bzip3

#69

Post by winwang »

teiferer wrote: post #32
amelius wrote: post #18

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. Sounds like perhaps a nice testcase for formalization + AI?

It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.

imo, partially because it's still not easy (in terms of code -> formal proof). With AI, I've been Lean-ifying a simpler (but non-trivial) algo. Pointing (current) AI at it only goes so far and in fact might go "too far" in certain cases, where a non-formalized argument would have sufficed. There's also "who watches the watcher" -- did it really prove what we're supposed to prove?

For something like these compression algos, though, I imagine it would be much easier since they already have actual proofs out there.

Re: bzip3

#70

Post by retrac »

8organicbits wrote: post #26

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…

For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
Post reply on HN