171–180 of 181 posts
Earlier quoted context omitted.
No it doesn’t? You can get zstd on macOS just fine. We’re talking about zstd, not bash, and supported releases not what ships with the distribution.
You can get bash4 or 5 too, as I already said. I can't help you with your depth of experience. I didn't even say zstd was unfit to depend on, just that it was a fair question.
Every deviation you make from stock is something else you have to put into. The onboarding docs and convince all your coworkers to install. They won’t. Or they think they will but will miss a step or machine. Or for language specific tools, they’ll use a multiversion tool like asdf or nvm or ram and have to reinstall again every upgrade.
It’s not up to me. It’s not up to you. It’s up to us. And if you can’t see that, maybe you shouldn’t be working with other developers. Or maybe you haven’t been and you need to.
Earlier quoted context omitted.
I think you would just need ANS, not the rest of zstd.
I don’t know enough to evaluate that, but it sounds plausible. Apparently modifying or integrating zstd into custom solutions was a common path in submissions to, at the very least, the GDCC 2021 T2 contest. This is all well outside of my learned competence so I’m just here to learn and ask naive questions.
As people mentioned elsewhere in the thread, Burrows-Wheeler isn’t really composable with other systems. Nobody has figured out a reasonable way to combine LZ77 and Burrows-Wheeler. That’s why zstd + bzip2 does not work. But you do need an entropy coding system to go with Burrows-Wheeler… and FSE fits the bill.
Earlier quoted context omitted.
My standard test is compressing a "dd" disc image of a Linux install (I use these for work), with unused blocks being zeroed. Results: Uncompressed: 7,516,192,768 zstd: 1,100,323,366 bzip3 -b 511 -j 4: 1,115,125,019
Hi, tool author here! Thank you for your benchmark! As you may be aware, different compression tools fill in different data type niches. In particular, less specialised statistical methods (bzip2, bzip3, PPMd) generally perform poorly on vaguely defined binary data due to unnatural distribution of the underlying data that at least in bzip3's case does not lend well to suffix sorting. Conversely, Lempel-Ziv methods us…
> Bzip3: A better and stronger spiritual successor to BZip2 Please, no. What's next ? BZip4 and BZip5, each one incompatible with each other ?
Earlier quoted context omitted.
zstd is faster and provides better compression than gzip at every point on the curve. There is no reason to use gzip these days other than backwards compatibility.
> bzip2 -9 -k all.tar 981.78s user 9.77s system 95% cpu 8M memory 17:16.64 total > bzip3 -e -b 256 -j 12 all.tar 2713.81s user 16.28s system 634% cpu 18301M memory 7:10.10 total The memory usage is one reason: 8M vs 18301M
I agree w other commenters, that Lzip would be a good baseline to compare it to, besides bzip2.
Nice job though on testing it on so many CPU architectures!
I still remember going crazy about bzip (the first one) and re-compressing all my data with it. And then I remember discovering, several years later, that bzip (the first one) is an obsolete format that is now difficult to even decompress. I learned my lesson and now use horribly sub-optimal formats that I'm sure will stick around for a long time, if not forever.
New frameworks, new languages, new cars, new heating systems.
If you want to have a stable solution, use "boring" things. They are boring for a reason; not much stuff is changing in them anymore. They are stable.
I use boring stuff all the time.
Earlier quoted context omitted.
I remember the lecturer commenting on what sort of sick and twisted mind could come up with such a ridiculous convoluted notion when I was taught it at university.
Wheeler was also one of the inventors of the "closed subroutine" AKA function, which had to be implemented via a hack as machines of the time did not include ISA support for "return": https://en.m.wikipedia.org/wiki/Wheeler_Jump
Earlier quoted context omitted.
> bzip2 -9 -k all.tar 981.78s user 9.77s system 95% cpu 8M memory 17:16.64 total > bzip3 -e -b 256 -j 12 all.tar 2713.81s user 16.28s system 634% cpu 18301M memory 7:10.10 total The memory usage is one reason: 8M vs 18301M
I think you intended to reply somewhere else. This isn't responsive.