Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lossless compression with Brotli

blogs.dropbox.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Lossless compression with Brotli

#21

Post by saynsedit »

monocasa wrote: post #14
saynsedit wrote: post #10

I'm having a hard time understanding the motivations for porting to rust... If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)

> If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. They can't block everything with SECCOMP, the code still has to be useful. It also must still do it's job correctly, which it won't if it's compromised. > If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway?…

Preventing compromise and ensuring determinism using checksums and SECCOMP is more reliable and easier to implement than reimplementing in rust an already de-facto secure library used millions of times every day.

I'm sorry but the Rust rationale just isn't there. Porting to Rust is a strictly less efficient way of accomplishing the same exact goal.

Now, writing a new library in rust, that's a different story and actually is reasonable.

Re: Lossless compression with Brotli

#22

Post by saynsedit »

daniel_rh wrote: post #19
Jweb_Guru wrote: post #16

Earlier quoted context omitted.

> If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. Not really. For instance, a compromised library can still rewrite the file you're storing in DropBox to contain different contents, which could ultimately result in remote code execution when you redownloaded it. Seccomp only makes sure the decompression server is safe from a rogue proce…

The file's sha256sum can be verified before the file is sent to any users, so there's no chance of RCE there, even with a hypothetical C brotli-- but a reproducible decompression is key. Additionally, if you want to do the decompression on client-side, facilities like SECCOMP simply may not be available on that platform. And in that case, having a language like Rust to guard against RCE is an excellent idea. Also it…

Is the brotli decompression step really the most dangerous vector on the client-side? What about all the non-verified client-side native code that actually interprets file data? From a practical perspective using the C code doesn't deteriorate existing conditions and using the Rust code doesn't improve them.

Re: Lossless compression with Brotli

#24

Post by JyrkiAlakuijala »

niftich wrote: post #7
maxst wrote: post #5

Somebody, make "image format powered by brotli", please.

Since most of Brotli's improvements over its ancestor LZ77 are due to its large, hardcoded, text-corpus dictionary [1], most of the algorithm's strengths would be wasted on binary data like images. Zopfli, from the same people, is a DEFLATE encoder, so it can be used in PNG [2] and this has already been added to some optimizers, e.g. AdvanceCOMP [3] [1] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e [2] https…

About 25 % of compression improvements for short files (such as web pages) come from the static dictionary. The rest are format improvements. The relatively small static dictionary does not improve the compression of long files.

Re: Lossless compression with Brotli

#25

Post by moosingin3space »

saynsedit wrote: post #10

I'm having a hard time understanding the motivations for porting to rust... If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)

> If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)

Yes, and now they can rely on the Rust compiler to help keep them safe whenever they make a change. If they had stayed in C, any changes would require more manual reasoning about uninitialized memory.

Re: Lossless compression with Brotli

#26

Post by vvanders »

Jweb_Guru wrote: post #15
Manishearth wrote: post #12

Earlier quoted context omitted.

It might be possible that jemalloc already does this? Rust uses it by default, and while I don't recall the specifics jemalloc does a bunch of things in userspace wrapped around malloc that make it better than malloc. Rust makes it easy to swap out the allocator too, so I'd love to see an allocator lib specifically focused on size management :)

Yes, this is precisely how jemalloc works.

Does jemalloc allow you to report telemetry and feed that back into the initial pool sizes? That's what makes the gamedev oriented allocators interesting.

FWIW a "Frame allocator" was always a block of memory where malloc returned pointer into the block and advanced the pointer. At the end of rendering a frame you just reset the pointer to the start of the block. The downside is you need to guarantee that nothing in has a lifetime longer than a frame, something Rust should be good at :).

I think what the GP is referring to is a pooling allocator that pools similar sized allocations.

Re: Lossless compression with Brotli

#27

Post by Jweb_Guru »

vvanders wrote: post #26
Jweb_Guru wrote: post #15

Earlier quoted context omitted.

Yes, this is precisely how jemalloc works.

Does jemalloc allow you to report telemetry and feed that back into the initial pool sizes? That's what makes the gamedev oriented allocators interesting. FWIW a "Frame allocator" was always a block of memory where malloc returned pointer into the block and advanced the pointer. At the end of rendering a frame you just reset the pointer to the start of the block. The downside is you need to guarantee that nothing in…

> Does jemalloc allow you to report telemetry and feed that back into the initial pool sizes? That's what makes the gamedev oriented allocators interesting.

Not AFAIK. There are a lot of optimizations it's quite difficult to do with dynamic pool sizes, which aren't worthwhile in the general case.

Re: Lossless compression with Brotli

#28

Post by saynsedit »

moosingin3space wrote: post #25
saynsedit wrote: post #10

I'm having a hard time understanding the motivations for porting to rust... If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)

> If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites) Yes, and now they can rely on the Rust compiler to help keep them safe whenever they make a change. If they had stayed in C, any changes would require more manual reasoning about uninitialized memory.

That's assuming this library will change often which I doubt since it's not upstream. Cost-benefit falls short here.

This "future changes" logic would apply better if this were an upstream library that were expected to change often enough to offset the upfront cost of porting an entire library.

Re: Lossless compression with Brotli

#29

Post by frewsxcv »

mmastrac wrote: post #2

Somewhat interesting for those who missed it - another rust/brotli project was a target for afl-rust. IIRC it didn't find any memory safety issues, but it definitely exposed a few panic crashes: https://github.com/frewsxcv/afl.rs (discussion https://news.ycombinator.com/item?id=11936983 ) I don't think they mentioned fuzzing their Brotli decompressor in this article, but I hope they try afl-rust.

(afl.rs maintainer here)

I ran AFL on rust-brotli for a week a couple weeks ago. It didn't find anything. I plan to try again soon! No one is safe from AFL.

Re: Lossless compression with Brotli

#30

Post by JoshTriplett »

frewsxcv wrote: post #29
mmastrac wrote: post #2

Somewhat interesting for those who missed it - another rust/brotli project was a target for afl-rust. IIRC it didn't find any memory safety issues, but it definitely exposed a few panic crashes: https://github.com/frewsxcv/afl.rs (discussion https://news.ycombinator.com/item?id=11936983 ) I don't think they mentioned fuzzing their Brotli decompressor in this article, but I hope they try afl-rust.

(afl.rs maintainer here) I ran AFL on rust-brotli for a week a couple weeks ago. It didn't find anything. I plan to try again soon! No one is safe from AFL.

Have you run AFL against git2-rs? I'd love to see that interface hammered on, to make sure it doesn't expose any unsafe behavior from the underlying library.
Post reply on HN