71–80 of 128 posts
Previously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.
Any relation to Iczelion from the masm32 tutorial?
Why LGPL when the original license is more permissive? ( https://sourceware.org/bzip2/manual/manual.html )
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.
It didn't work, or the gains were so marginal that I ended up scrapping the approach.
So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
Earlier quoted context omitted.
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive…
Earlier quoted context omitted.
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
Earlier quoted context omitted.
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.