Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bzip3: A spiritual successor to BZip2

github.com

Post reply on HN
Print view View original

41–50 of 181 posts

Re: Bzip3: A spiritual successor to BZip2

#41

Post by cwillu »

lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

Simple enough to be safe, at the cost of performance: uncompress and compare to the original.

Re: Bzip3: A spiritual successor to BZip2

#42

Post by cowsandmilk »

lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

I mean, hosting downloads that have been compressed with this seems fine. I have the original, I just want a smaller version for those downloading.

Re: Bzip3: A spiritual successor to BZip2

#43

Post by hinkley »

lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

Honestly I think this may be part of why compression in flight has been a growth area in compression lately. If your transport encoding becomes garbled people notice immediately. If you manage to serve a few billion compressed requests without any incident reports then maybe you can trust it for data at rest.

Re: Bzip3: A spiritual successor to BZip2

#44

Post by lexicality »

cwillu wrote: post #41
lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

Simple enough to be safe, at the cost of performance: uncompress and compare to the original.

And what do you do if it doesn't match?

Re: Bzip3: A spiritual successor to BZip2

#45

Post by hinkley »

cwillu wrote: post #41
lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

Simple enough to be safe, at the cost of performance: uncompress and compare to the original.

You could have bugs that show up on different hardware or compiler versions. So the round trip is table stakes but not a guarantee.

Edit: someone deleted a response that said that if you can read it back then the data is there. I think in a data recovery sense that’s definitely true, if it’s consistent across inputs. But building something that simulates the undefined behavior or race condition - if it’s symmetrical between read and write could be pretty tricky. And you’d have to decode based on file version number to know if you need the emulation. So possible but terrible to maintain, and the interim versions from creating and discovering the bug would still be broken.

Re: Bzip3: A spiritual successor to BZip2

#46

Post by PaulKeeble »

lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

I had bzip2 eat some files some years back and ended up loosing them. I don't trust bzip2 for backup files and I wouldn't trust bzip3 either as a result.

Re: Bzip3: A spiritual successor to BZip2

#47

Post by supertrope »

lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

A shell script to run a checksum on the original file, compress, uncompress, and verify output integrity should be trivial.

Re: Bzip3: A spiritual successor to BZip2

#48

Post by jrockway »

loeg wrote: post #32
ThatGuyRaion wrote: post #28

Earlier quoted context omitted.

zstd is a great setup, for sure, but the build system they use is patently awful. Can someone make an autoconf or hell, even CMake build system for them pleasee???

Just install the binary package from whatever distro you use? Why do you need to build it? But if it matters, FreeBSD has a pretty trivial BSDmake build of it: https://github.com/freebsd/freebsd-src/blob/main/lib/libzstd... https://github.com/freebsd/freebsd-src/blob/main/usr.bin/zst... You could easily do something similar in GNU make or whatever without the dependencies on the FBSD build system. It's basically just…

I suppose the point of open source software is to edit the source code and make your own version. If you can't build it, you can't do that.

Re: Bzip3: A spiritual successor to BZip2

#49

Post by jrockway »

idoubtit wrote: post #25
loeg wrote: post #22

Earlier quoted context omitted.

zstd is faster and provides better compression than gzip at every point on the curve. There is no reason to use gzip these days other than backwards compatibility.

I've reached the same conclusion, and I've been using zstd unless I want extremely fast (de)compression with lz4. And even when I still have to use gzip (the format), the executables of libdeflate are fully compatible and noticeably faster than gzip/gunzip (the tool).

This was on HN a while ago https://jolynch.github.io/posts/use_fast_data_algorithms/ and reaches the same conclusion. I am inclined to use Zstd even in cases where people would use lz4. Zstd is very fast.

Re: Bzip3: A spiritual successor to BZip2

#50

Post by Arech »

supertrope wrote: post #47
lexicality wrote: post #39

> DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE. I know every open source project (and quite a lot of expensive proprietary ones!) come with a "btw this software might wipe your computer, if it does that's your fault lol" clause in their license but I can't imagine trying to convince anyone else that using this for an…

A shell script to run a checksum on the original file, compress, uncompress, and verify output integrity should be trivial.

I wonder why this isn't inbuilt into the program..
Post reply on HN