91–100 of 133 posts
"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE." So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.
Earlier quoted context omitted.
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive…
Because JSON is an inefficient text encoding, compression (with custom dictionary) are likely to really well on those.
Books have recurring words, but probably much less.
Earlier quoted context omitted.
Not really, it's a popular dictionary-based compression format.
I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things. Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
If your platform/sdk/browser/standard-library comes with zlib/gzip/.. then it's often easier to just pick that.
No new dependencies is always a win. App size. Security, etc.
Otherwise, if zstd is easy to add, IMO I would always prefer, zstd, lz4 or brotli.
Earlier quoted context omitted.
> 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor. That always seemed annoying to me. They couldn't allocate 5 more bits so…
2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.
If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.
Previously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
> “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3 FWIW, the Burrows-Wheelers transform is also used by bzip2, so this isn’t a new feature even though that quote kinda sounds like it is.
https://arxiv.org/abs/1502.05937
https://github.com/nicolaprezza/lz-rlbwt
Which leads me to wonder if someone did, in fact, solve how to ‘combine zstd and bzip’ (so to speak) since the last time.
bzip3 not a terribly impressive or useful compression algo.
benchmark on enwik: https://www.mattmahoney.net/dc/text.html#1703
Earlier quoted context omitted.
I have an anecdote about compressed data. When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog. Imagine how fast me and th…
Solaris has had so many cool features, like ZFS or doors. What I liked about ZFS is you coul make snapshot, which is basically the solution to how to treat data files a single, cheap to access unit, yet still use standard apis for file management, great for containerizing apps, making copies for experiments, or shipping stuff. Node.js just received this as a bespoke, app-level feature. But these things are too many t…
Previously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
> “bzip3 is not yet listed on the large text compression benchmark” It is now And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.
On my desktop, decompressing with zstd requires 128 MB and 1 s, while bzip3 uses 3.2 GB and 90 s.
Without `-b 511`, bzip3's ratio decreases and decompression requires about the same memory as zstd. But the decompression is still two orders of magnitude slower.
For data recovery of archival stuff it's better to keep it uncompressed, no?
Although you do have a point that the compressed data might be more difficult to decipher, if it doesn't have sufficient redundancy to skip bad parts, or if it is essential that the data is aligned in a certain way (e.g. disk images, and probably many other formats) and the format doesn't take this into account. Shorter window sizes, window reset markers, and explicit offset information could mitigate those problems.