81–90 of 133 posts
Earlier quoted context omitted.
Not really, it's a popular dictionary-based compression format.
I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things. Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
Earlier quoted context omitted.
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
How did I miss zstd? Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults. gzip 7.3% 21s 9s bzip2 4.6% 251s 50s bzip3 3.3% 82s 69s zstd 6.9% 2s 3s lzma 4.7% 51s 3s
Previously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
> “bzip3 is not yet listed on the large text compression benchmark” It is now And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.
For data recovery of archival stuff it's better to keep it uncompressed, no?
Earlier quoted context omitted.
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
Comparing it to memcpy is idiotic.
Earlier quoted context omitted.
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive…
When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog.
Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left.
Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…
zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems. I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json
Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software…
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) l…
Earlier quoted context omitted.
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs. So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive…
I have an anecdote about compressed data. When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog. Imagine how fast me and th…
Sun was a really cool company.
\[T]/