Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lossless compression with Brotli

blogs.dropbox.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Lossless compression with Brotli

#51

Post by tveita »

saynsedit wrote: post #21
monocasa wrote: post #14

Earlier quoted context omitted.

> If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. They can't block everything with SECCOMP, the code still has to be useful. It also must still do it's job correctly, which it won't if it's compromised. > If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway?…

Preventing compromise and ensuring determinism using checksums and SECCOMP is more reliable and easier to implement than reimplementing in rust an already de-facto secure library used millions of times every day. I'm sorry but the Rust rationale just isn't there. Porting to Rust is a strictly less efficient way of accomplishing the same exact goal. Now, writing a new library in rust, that's a different story and actu…

People were finding buffer overflows in zlib when it was more than twice as old as Brotli, and they're still finding bugs in libpng. I don't see any reason to think that Brotli is "de-facto secure" or even mature. If history is an indication it takes decades to shake the buffer overflows out of even a very popular C library.

Re: Lossless compression with Brotli

#53

Post by [deleted] »

tveita wrote: post #51
saynsedit wrote: post #21

Earlier quoted context omitted.

Preventing compromise and ensuring determinism using checksums and SECCOMP is more reliable and easier to implement than reimplementing in rust an already de-facto secure library used millions of times every day. I'm sorry but the Rust rationale just isn't there. Porting to Rust is a strictly less efficient way of accomplishing the same exact goal. Now, writing a new library in rust, that's a different story and actu…

People were finding buffer overflows in zlib when it was more than twice as old as Brotli, and they're still finding bugs in libpng. I don't see any reason to think that Brotli is "de-facto secure" or even mature. If history is an indication it takes decades to shake the buffer overflows out of even a very popular C library.

[deleted]

Re: Lossless compression with Brotli

#54

Post by saynsedit »

staticassertion wrote: post #50
saynsedit wrote: post #10

I'm having a hard time understanding the motivations for porting to rust... If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. If reasoning about uninitialized memory would take a review of the entire brotli code base, didn't the rust port require that anyway? (speaking as someone who has done a couple of cross-language rewrites)

> If they were going to run the whole thing in a SECCOMP container anyway, there is little damage a compromised C library could do. True, especially since they appear to be using a traditional seccomp sandbox, which means the process really can't do much. Based on the architecture there is no broker process involved, you simply feed in data via read() and it writes the data out via write(). It has access to two other…

Binaries in a SECCOMP cannot by any means read/write arbitrary files. They would need access to open(), socket(), etc. there is only read(), write(), exit() and sigreturn(). Only the intentionally inherited file descriptors are accessible.

As for your "kernel attack surface is lowered" argument, it's incredibly silly. If read() and write() ever had a vulnerability we have much bigger problems than binaries in a SECCOMP. It's like saying SHA256 isn't secure because there is technically a risk of collision.

Determinism and robustness aren't valid, they can use checksums to verify data coming out of the SECCOMP container. Same effect, much less effort.

And for your argument against de facto security: security reasoning isn't binary. It's a risk based assessment. Nothing is 100% secure and context matters. Binary thinking and dogma are incredibly dangerous. Spouting "Rust is secure" is dogma, it's not true. Rust is "99%" secure, unknown unknowns will always be out there.

In this case, context is important. My point isn't that c-brotli is as secure as rust-brotli. My point is that using c-brotli doesn't introduce any significant new security risk on the client-side given that there are lots of other c-based libraries already interpreting file data. Given that context, reimplementing brotli in rust for the purposes of increased security only marginally accomplishes its goal. It's like optimizing the part of your program that takes the least time, lots of effort for almost no gain.

On the server side, there's no gain at all to porting c-brotli to rust since you can already use SECCOMP and checksums. It's just wasted effort.

Am I saying there is no valid reason to port things to Rust? No. I'm saying it's invalid to port a library to rust for security reasons if you already have effective means of mitigating binary attacks, I.e. SECCOMP and checksums.

Re: Lossless compression with Brotli

#55

Post by saynsedit »

daniel_rh wrote: post #32
saynsedit wrote: post #21

Earlier quoted context omitted.

Preventing compromise and ensuring determinism using checksums and SECCOMP is more reliable and easier to implement than reimplementing in rust an already de-facto secure library used millions of times every day. I'm sorry but the Rust rationale just isn't there. Porting to Rust is a strictly less efficient way of accomplishing the same exact goal. Now, writing a new library in rust, that's a different story and actu…

Your argument is certainly reasonable. One case where it might matter is if SECCOMP weren't available on all platforms that needed to decompress data. Also: a vulnerability could decide to only strike after a certain clock date. In that way it could sneak past certain checksum checks and still cause issues later.

On the client side there is already a vast amount of C-code that is interpreting file data, the brotli portion is a minimal part of that security risk. Porting brotli to rust for the purposes of increased security is like optimizing the part of your program that takes the least time. Lots of effort for minimal gain.

Re: Lossless compression with Brotli

#56

Post by emn13 »

blank_state wrote: post #49
JyrkiAlakuijala wrote: post #37

Earlier quoted context omitted.

Brotli's fastest compression is slightly faster than zstd's. Zstd decompresses faster, but neither is slow. Zstd can use sliding window size longer than 16 MB, in brotli this is limited to 16 MB to have guarantees of the maximum resource use at decoding time. Zstd's longer sliding window helps with the longest files (16 MB+), and often benchmarking is done with 100 MB or even 1 GB files. Brotli compresses usually mor…

> Brotli's fastest compression is slightly faster than zstd's. Come on, this is not serious. Brotli's fastest compression algorithm is still significantly slower than zstd. And more importantly, it compresses _much worse_. For a 3rd party evaluation, one can try [TurboBench]( https://github.com/powturbo/TurboBench ) or even [lzbench]( https://github.com/inikep/lzbench ) which are open-sourced. Squash introduces a wra…

The two benchmarks you mention aren't very useful for comparing brotli and zstd, because they deal with large datasets. Per-call overhead matters a lot for small files, and I imagine (particularly brotli) is aimed at small files. Zstd (usefully!) calls out dictionary compression, which hints that small files matter for it too, but I'm not positive there's any specific use case for zstd in mind.

In any case, neither the compression ratios nor the speed of large-file compression necessarily say much about small file performance. There's just much more context to search in a 100MB file than there is in a 10KB file.

Having said that, there's no reason to assume brotli is better for small files; there's just no way to tell given the links you provide.

I'm not affiliated with nor use neither zstd nor brotli.

Re: Lossless compression with Brotli

#57

Post by saynsedit »

tveita wrote: post #51
saynsedit wrote: post #21

Earlier quoted context omitted.

Preventing compromise and ensuring determinism using checksums and SECCOMP is more reliable and easier to implement than reimplementing in rust an already de-facto secure library used millions of times every day. I'm sorry but the Rust rationale just isn't there. Porting to Rust is a strictly less efficient way of accomplishing the same exact goal. Now, writing a new library in rust, that's a different story and actu…

People were finding buffer overflows in zlib when it was more than twice as old as Brotli, and they're still finding bugs in libpng. I don't see any reason to think that Brotli is "de-facto secure" or even mature. If history is an indication it takes decades to shake the buffer overflows out of even a very popular C library.

Defacto secure doesn't mean "has no buffer overflows." It just means it's already a prevalent security risk outside of your system.

Re: Lossless compression with Brotli

#58

Post by niftich »

est wrote: post #46
niftich wrote: post #7

Earlier quoted context omitted.

Since most of Brotli's improvements over its ancestor LZ77 are due to its large, hardcoded, text-corpus dictionary [1], most of the algorithm's strengths would be wasted on binary data like images. Zopfli, from the same people, is a DEFLATE encoder, so it can be used in PNG [2] and this has already been added to some optimizers, e.g. AdvanceCOMP [3] [1] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e [2] https…

First time saw the Brotli dictionary. It has duplicates. Line 3131 and 8704 both are "操作"

Interestingly, there are also 121 different transformations [1] you can apply to each dictionary word, from adding various prefixes and suffixes, trimming letters, and some more complex ones [2].

If the plain-text dictionary linked earlier [3] is accurate, it'd appear that the dictionary contains a lot of redundant forms.

[1] https://tools.ietf.org/html/draft-alakuijala-brotli-11#page-... [2] https://tools.ietf.org/html/draft-alakuijala-brotli-11#appen... [3] https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e

Re: Lossless compression with Brotli

#60

Post by JyrkiAlakuijala »

blank_state wrote: post #49
JyrkiAlakuijala wrote: post #37

Earlier quoted context omitted.

Brotli's fastest compression is slightly faster than zstd's. Zstd decompresses faster, but neither is slow. Zstd can use sliding window size longer than 16 MB, in brotli this is limited to 16 MB to have guarantees of the maximum resource use at decoding time. Zstd's longer sliding window helps with the longest files (16 MB+), and often benchmarking is done with 100 MB or even 1 GB files. Brotli compresses usually mor…

> Brotli's fastest compression is slightly faster than zstd's. Come on, this is not serious. Brotli's fastest compression algorithm is still significantly slower than zstd. And more importantly, it compresses _much worse_. For a 3rd party evaluation, one can try [TurboBench]( https://github.com/powturbo/TurboBench ) or even [lzbench]( https://github.com/inikep/lzbench ) which are open-sourced. Squash introduces a wra…

TurboBench didn't compile straigth after git clone. lzbench did:

I get a test file by: wget https://web.archive.org/web/20151222062543/http://www.micros...

The test file is 267253 bytes.

$ ./lzbench -ebrotli,0,1,2,5,7,9,11/zstd,1,22 testfile

brotli 0.4.0 -0 compresses 783 MB/s and decompresses 809 MB/s

zstd 0.7.1 -1 compresses 586 MB/s and decompresses 1691 MB/s

brotli 0.4.0 -7 compresses 57 MB/s, decompresses 873 MB/s to 28185 bytes

brotli 0.4.0 -11 compresses to 25413 bytes

zstd 0.7.1 -22 compresses in 4.01 MB/s to 28363 bytes

Of course it is an unfair example because of the static dictionary that brotli uses, but it is not a pathological example: Thai is not part of the static dictionary. The numbers are on a i7-4790K@4.00 GHz.

Brotli's fastest compression is faster than that of zstd, at least as shown with lzbench and this file. Also brotli wins in compression density. In this file the win is 10.5 % less bytes for brotli -11 than for zstd -22.

Post reply on HN