Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

71–80 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#71

Post by orangeboats »

tlb wrote: post #70
orangeboats wrote: post #68

Earlier quoted context omitted.

Can't the decompressor still produce a malicious uncompressed file?

Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.

Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo.

However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a malicious uncompressed file from a known good compressed file.

Re: OpenZL: An open source format-aware compression framework

#72

Post by yorwba »

orangeboats wrote: post #71
tlb wrote: post #70

Earlier quoted context omitted.

Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.

Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo. However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a mali…

If the decompressor is included in the compressed file and it's malicious, the file can hardly be called known good.

Re: OpenZL: An open source format-aware compression framework

#73

Post by mort96 »

orangeboats wrote: post #71
tlb wrote: post #70

Earlier quoted context omitted.

Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.

Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo. However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a mali…

The format-specific decompressor is part of the compressed file. Nothing here crosses a security boundary. Either the compressed file is trustworthy and therefore decompresses into a trustworthy file, or the compressed file is not trustworthy and therefor decompresses into a non-trustworthy file.

If the compressed file is malicious, it doesn't matter whether it's malicious because it originated from a malicious uncompressed file, or is malicious because it originated from a benign uncompressed file and the transformation into a compressed file introduces the malicious parts due to the bundled custom decompressor.

Re: OpenZL: An open source format-aware compression framework

#74

Post by tecleandor »

yorwba wrote: post #72
orangeboats wrote: post #71

Earlier quoted context omitted.

Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo. However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a mali…

If the decompressor is included in the compressed file and it's malicious, the file can hardly be called known good .

But also I guess the logic of the decompressor could output different files in different occasions, for example, if it detects a victim, making it difficult to verify.

Re: OpenZL: An open source format-aware compression framework

#75

Post by TiredOfLife »

magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

And no mention of zpaq that has had emedable decompressors feature for 15 years

Re: OpenZL: An open source format-aware compression framework

#76

Post by lifthrasiir »

magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

As someone seriously trying to develop a compressed archive format with WebAssembly, sandboxing is actually easy and that's indeed why WebAssembly was chosen. The real problem is determinism, which WebAssembly does technically support but actual implementations may vary significantly. And even when WebAssembly can be made fully deterministic, function calls made to those WebAssembly modules may still be undeterministic! I tried very hard to avoid such pitfalls in my design, and it is entirely reasonable to avoid WebAssembly due to these issues.

Re: OpenZL: An open source format-aware compression framework

#78

Post by viraptor »

tecleandor wrote: post #74
yorwba wrote: post #72

Earlier quoted context omitted.

If the decompressor is included in the compressed file and it's malicious, the file can hardly be called known good .

But also I guess the logic of the decompressor could output different files in different occasions, for example, if it detects a victim, making it difficult to verify.

If it can "detect a victim", then the sandbox is faulty. The decompressor shouldn't see any system details. Only the input and output streams.

Re: OpenZL: An open source format-aware compression framework

#79

Post by jo-m »

themerone wrote: post #63
snapplebobapple wrote: post #62

Earlier quoted context omitted.

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

Wasm can be sandboxed. Its a safe as visiting a website with javascript.

So, not very safe.

Re: OpenZL: An open source format-aware compression framework

#80

Post by slmkbh »

fnands wrote: post #77
fnands wrote: post #10

Cool, but what's the Weissman Score?

Alright, Silicon Valley references are not popular on HN it seems.

Lack of self irony... I was also looking for this :)

Having just re watched the show, it is remarkable how little changed for the better...

Post reply on HN