Viewing profile — felixhandte

SDDL (and the front-end task of reshaping data in general) is only one component of OpenZL. Once you have the streams, you can do all sorts of transformations to them that Zstd doe…

The OpenZL core supports arbitrary composition of graphs. So you can do this now via the compressor construction APIs. We just have to figure out how to make it easy to do.

Wanna hop over to https://github.com/facebook/openzl/issues/76 ?

Update: let's continue discussing genomic sequence compression on https://github.com/facebook/openzl/issues/76 .

Yes. None of the algorithms under test used any hardware acceleration in the benchmarks we ran.

Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descript…

Yes, definitely! Chunking support is currently in development. Streaming and seeking and so on are features we will certainly pursue as we mature towards an eventual v1.0.0.

Thanks @dang!

It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transform…

Let us know how it goes! We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people w…

In addition to open-sourcing the code today, we also published: - A white paper: https://arxiv.org/abs/2510.03203 - A blog post: https://engineering.fb.com/2025/10/06/developer-too…

In addition to the blog post, here are the other things we've published today: Code: https://github.com/facebook/openzl Documentation: https://openzl.org/ White Paper: https://arxi…

Zstd has a similar-ish capability called "repetition codes" [0]. The first stage of Zstd does LZ77 matching, which transforms the input into "sequences", a series of instructions e…

This is because Zstd's long-distance matcher looks for matching sequences of 64 bytes [0]. Because long matching sequences of the data will likely have the newlines inserted in dif…

Awesome work! I get exactly that green cast and muted color range off of my flatbed scans (Epson v800). This is a really intriguing path to fixing them I hadn't considered. It seem…

That would only work in the case that the camera is fixed on a tripod and has a long period of stable / rigid pointing before the exposure during which to collect this data. This i…

Probably referring to this: https://futurism.com/the-byte/piece-falls-off-boeing-starlin...

You're saying that a system that can recognize flaws in the alignment imposed on it can reject that alignment, but that doesn't follow. Sure, humans act against their own interests…

Sure, but Gödel encoding is pretty much purely a theoretical exercise. I'm not sure anyone anywhere has ever practically manipulated Gödel-encoded expressions in a useful way. His …

1. There are several implementations of Zstandard other than the "reference" libzstd that we maintain, tagged as "Ports" in the table here [0]. You are of course welcome to create …

Indeed--at this point zstd is the product of something like 25 man-years of improvements and optimizations.

What differences do you have in mind? Zstd is largely compatible as a drop-in replacement for those commands. There are differences, but it's our goal to minimize them.

There's also , which I (en-US) call "angle brackets".