Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

bzip3

github.com

Post reply on HN
Print view View original

31–40 of 130 posts

Re: bzip3

#31

Post by dzaima »

nvme0n1p1 wrote: post #29
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits so…

> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)

Re: bzip3

#32

Post by teiferer »

amelius wrote: post #18

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. Sounds like perhaps a nice testcase for formalization + AI?

It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.

Re: bzip3

#33

Post by benatkin »

nvme0n1p1 wrote: post #30
8organicbits wrote: post #26

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

Not really, it's a popular dictionary-based compression format.

Re: bzip3

#34

Post by esseph »

benatkin wrote: post #33
nvme0n1p1 wrote: post #30

Earlier quoted context omitted.

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

Not really, it's a popular dictionary-based compression format.

”Not really" what?

It's hard to understand what point you're trying to make. Can you clarify?

Re: bzip3

#35

Post by rmunn »

nvme0n1p1 wrote: post #29
ot wrote: post #12

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty m…

> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits so…

2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.

Re: bzip3

#36

Post by sergiotapia »

"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE."

So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

Re: bzip3

#38

Post by [deleted] »

amelius wrote: post #18

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. Sounds like perhaps a nice testcase for formalization + AI?

[deleted]

Re: bzip3

#39

Post by fmx »

sergiotapia wrote: post #36

"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE." So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.
Post reply on HN