Lossless compression with Brotli

#1

Post by nuriaion » Wed, Jun 29, 2016, 2:31 PM UTC

Lossless compression with Brotli
blogs.dropbox.com

Re: Lossless compression with Brotli

#2

Post by mmastrac » Wed, Jun 29, 2016, 5:03 PM UTC

Somewhat interesting for those who missed it - another rust/brotli project was a target for afl-rust. IIRC it didn't find any memory safety issues, but it definitely exposed a few panic crashes:

https://github.com/frewsxcv/afl.rs (discussion https://news.ycombinator.com/item?id=11936983)

I don't think they mentioned fuzzing their Brotli decompressor in this article, but I hope they try afl-rust.

Re: Lossless compression with Brotli

#3

Post by vvanders » Wed, Jun 29, 2016, 5:10 PM UTC

Awesome to see the contributed back their custom allocator that lets you allocate from a fixed block: https://github.com/dropbox/rust-alloc-no-stdlib

Solid stuff from the looks of it.

Re: Lossless compression with Brotli

#4

Post by sanxiyn » Wed, Jun 29, 2016, 5:11 PM UTC

mmastrac wrote: post #2

Somewhat interesting for those who missed it - another rust/brotli project was a target for afl-rust. IIRC it didn't find any memory safety issues, but it definitely exposed a few panic crashes: https://github.com/frewsxcv/afl.rs (discussion https://news.ycombinator.com/item?id=11936983 ) I don't think they mentioned fuzzing their Brotli decompressor in this article, but I hope they try afl-rust.

Apparently this(https://github.com/dropbox/rust-brotli) is much faster than the one(https://github.com/ende76/brotli-rs) fuzzed by afl.rs. Like 10x faster. See https://github.com/ende76/brotli-rs/issues/24.

Re: Lossless compression with Brotli

#5

Post by maxst » Wed, Jun 29, 2016, 5:17 PM UTC

Somebody, make "image format powered by brotli", please.

Re: Lossless compression with Brotli

#6

Post by sanxiyn » Wed, Jun 29, 2016, 5:21 PM UTC

maxst wrote: post #5

Somebody, make "image format powered by brotli", please.

I think zstd will be better: https://github.com/Cyan4973/zstd.

Re: Lossless compression with Brotli

#7

Post by niftich » Wed, Jun 29, 2016, 5:27 PM UTC

maxst wrote: post #5

Somebody, make "image format powered by brotli", please.

Since most of Brotli's improvements over its ancestor LZ77 are due to its large, hardcoded, text-corpus dictionary [1], most of the algorithm's strengths would be wasted on binary data like images.

Zopfli, from the same people, is a DEFLATE encoder, so it can be used in PNG [2] and this has already been added to some optimizers, e.g. AdvanceCOMP [3]

[1] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e [2] https://github.com/google/zopfli/commit/337d27f25ef15a6cf34f... [3] http://www.advancemame.it/doc-advpng.html

Re: Lossless compression with Brotli

#8

Post by mozumder » Wed, Jun 29, 2016, 5:37 PM UTC

Can Brotli compress streamed blocks, with SYNC_FLUSH or FULL_FLUSH the way zlib can?

Re: Lossless compression with Brotli

#9

Post by gregwtmtno » Wed, Jun 29, 2016, 5:43 PM UTC

vvanders wrote: post #3

Awesome to see the contributed back their custom allocator that lets you allocate from a fixed block: https://github.com/dropbox/rust-alloc-no-stdlib Solid stuff from the looks of it.

That's going to be really great for embedded projects.

Re: Lossless compression with Brotli

#10

Post by saynsedit » Wed, Jun 29, 2016, 6:05 PM UTC

I'm having a hard time understanding the motivations for porting to rust...

If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do.

If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)