Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

61–70 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#61

Post by fwip »

terrelln wrote: post #52
bede wrote: post #50

Earlier quoted context omitted.

For BAM this could be a good place to start: https://www.htslib.org/benchmarks/CRAM.html Happy to discuss further

Amazing, thank you! I will take a look as soon as I get a chance. Looking at the BAM format, it looks like the tokenization portion will be easy. Which means I can focus on the compression side, which is more interesting.

Another format that might be worth looking at in the bioinformatics world is hdf5. It's sort of a generic file format, often used for storing multiple related large tables. It has some built-in compression (gzip IIRC) but supports plugins. There may be an opportunity to integrate the self-describing nature of the hdf5 format with the self-describing decompression routines of openZL.

Re: OpenZL: An open source format-aware compression framework

#62

Post by snapplebobapple »

magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

Re: OpenZL: An open source format-aware compression framework

#63

Post by themerone »

snapplebobapple wrote: post #62
magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

Wasm can be sandboxed. Its a safe as visiting a website with javascript.

Re: OpenZL: An open source format-aware compression framework

#64

Post by pabs3 »

felixhandte wrote: post #55
viraptor wrote: post #53

I wonder, given the docs, how well could AI translate imhex and Kaitai descriptions into SDDL. We could get a few good schemas quickly that way.

Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descriptions. I'll check them out.

There are a ton of these. GNU Poke comes to mind too.

Re: OpenZL: An open source format-aware compression framework

#65

Post by porridgeraisin »

This method reminds me of how deep learning models get compressed for deployment on accelerators. You take advantage of different redundancies of different data structures and compress each of them using a unique method.

Specifically the dictionary + delta-encoded + huffman'd index lists method mentioned in TFA, is commonly used for compressing weights. Weights tend to be sparse, but clustered, meaning most offsets are small numbers with the occasional jump, which is great for huffman.

Re: OpenZL: An open source format-aware compression framework

#66

Post by unsigner »

felixhandte wrote: post #2

In addition to the blog post, here are the other things we've published today: Code: https://github.com/facebook/openzl Documentation: https://openzl.org/ White Paper: https://arxiv.org/abs/2510.03203

Congrats on the release. I was wondering what the zstd team is up to lately.

You mentioned something about grid structured data being in the plans - can you give more details?

Have you done experiments with compressing BCn GPU texture formats? They have a peculiar branched structure, with multiple sub formats packed tightly in bitfields of 64- or 128-bit blocks; due to the requirement of fixed ratio and random access by the GPU they still leave some potential compression on the table.

Re: OpenZL: An open source format-aware compression framework

#68

Post by orangeboats »

themerone wrote: post #63
snapplebobapple wrote: post #62

Earlier quoted context omitted.

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

Wasm can be sandboxed. Its a safe as visiting a website with javascript.

Can't the decompressor still produce a malicious uncompressed file?

Re: OpenZL: An open source format-aware compression framework

#70

Post by tlb »

orangeboats wrote: post #68
themerone wrote: post #63

Earlier quoted context omitted.

Wasm can be sandboxed. Its a safe as visiting a website with javascript.

Can't the decompressor still produce a malicious uncompressed file?

Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.
Post reply on HN