Viewing profile — terrelln

Fusion is so useful though. It is easy to set up on ships because it requires no consumables other than the fuel cells, and it has a small footprint. It is easy to set up power on …

I ran into a fun crash a year or so ago in the interaction of clang’s profile guided speculative devirtualization and identical code folding (ICF) done by BOLT on the binary. Clang…

> Also making good progress on getting a slimmer version of zstd into the stdlib Awesome! Please let me know if there is anything I can do to help

Yeah, it isn't quite that simple. E.g. `/bin/ksh` reports 1.4MB, but it is actually 2.4MB. Initially, I thought it was because the file was sparse, but there are only 493KB of zero…

I've figured out the issue. Use `wc -c` instead of `du`. I can repro on my Mac with these steps with either `zstd` or `gzip`: $ rm -f ksh.zst $ zstd ksh.zst $ du -h ksh.zst 1.2M ks…

Ah I understand. In this benchmark, Zstd's decompression time is 284 MB/s, and Gzip's is 330 MB/s. This benchmark is likely dominated by file IO for the faster decompressors. On th…

Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of …

Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This see…

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file …

Its slang that has made its way from queer culture into mainstream. It is not meant to satisfy anyones ego, other than it just has vague positive connotations. It is used in a pret…

Out of curiosity, what was the input file format? We actually worked on a demo WAV compressor a while back. We are currently missing codecs to run the types of predictors that FLAC…

Amazing, thank you! I will take a look as soon as I get a chance. Looking at the BAM format, it looks like the tokenization portion will be easy. Which means I can focus on the com…

You could have an LLM generate the SDDL description [0] for you, or even have it write a C++ or Python tokenizer. If compression succeeds, then it is guaranteed to round trip, as t…

The charts in the "Results With OpenZL" section compare against all levels of zstd, xz, and zlib. On highly structured data where OpenZL is able to understand the format, it blows …

We left it out of the paper because it is an implementation detail that is absolutely going to change as we evolve the format. This is the function that actually does it [0], but t…

Do you happen to have a pointer to a good open source dataset to look at? Naively and knowing little about CRAM, I would expect that OpenZL would beat Zstd handily out of the box, …

You'd have to tell OpenZL what your format looks like by writing a tokenizer for it, and annotating which parts are which. We aim to make this easier with SDDL [0], but today is no…

> Is openzl indexable Not today. However, we are considering this as we are continuing to evolve the frame format, and it is likely we will add this feature in the future.

Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in …

There's a Quick Start guide here: https://openzl.org/getting-started/quick-start/ However, OpenZL is different in that you need to tell the compressor how to compress your data. Th…

Yeah, backend compression in columnar data formats is a natural fit for OpenZL. Knowing the data it is compressing is numeric, e.g. a column of i64 or float, allows for immediate w…

https://github.com/facebook/openzl https://arxiv.org/abs/2510.03203 https://openzl.org/

You can also use general purpose compressors like Zstandard to create a generic patch: zstd --patch-from old.json new.json -o patch.zst zstd --patch-from old.json -d patch.zst -o u…

Meta drove the Zstandard content encoding, but Google drove the adoption of Zstandard in Chrome. The faster Brotli levels could probably be made to match Zstandard’s compression sp…

The latest Zstandard exposes several parameters which are useful for reducing time to first byte latency in web compression. They make Zstandard cut the compressed data into smalle…