Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.
[deleted]
Broccoli: Syncing Faster by Syncing Less
21–30 of 58 posts
Re: Broccoli: Syncing Faster by Syncing Less
#22Surprised they didn't look more at zstd.
IME it's faster than brotli and often has a better compression ratio.
Re: Broccoli: Syncing Faster by Syncing Less
#23Surprised they didn't look more at zstd. IME it's faster than brotli and often has a better compression ratio.
It looks like they did, but having an implementation in a memory-safe language was one of their requirements. Learning that was for me the most fascinating part of the article.
Re: Broccoli: Syncing Faster by Syncing Less
#24Surprised they didn't look more at zstd. IME it's faster than brotli and often has a better compression ratio.
From the blog:
> Pre-coding: Since most of the data residing in our persistent store, Magic Pocket, has already been Brotli compressed using Broccoli, we can avoid recompression on the download path of the block download protocol. These pre-coded Brotli files have a latency advantage, since they can be delivered directly to clients, and a size advantage, since Magic Pocket contains Brotli codings optimized with a higher compression quality level.
Re: Broccoli: Syncing Faster by Syncing Less
#25Surprised they didn't look more at zstd. IME it's faster than brotli and often has a better compression ratio.
It looks like they did, but having an implementation in a memory-safe language was one of their requirements. Learning that was for me the most fascinating part of the article.
Surely Dropbox would have the engineering power to re-implement zstd in a memory safe language if it was sufficiently beneficial.
Re: Broccoli: Syncing Faster by Syncing Less
#26Is it possible to use this as rsync replacement ?
Re: Broccoli: Syncing Faster by Syncing Less
#27Surprised they didn't look more at zstd. IME it's faster than brotli and often has a better compression ratio.
We heavily investigated zstd and met with the brilliant inventor, Yann, who provided amazing insights into the design and rationale behind zstd and why it is so fast and such an amazing technology. I also recompiled zstd into rust using https://github.com/immunant/c2rust and tried using various webasm mechanisms to run it (I didn't get webasm quite fast enough, and teaching c2rust to make it safe would be quite a slog).
But the main reason we settled on Brotli was the second order context modeling, which makes a substantial difference in the final size of files stored on Dropbox (several percent on average as I recall, with some files getting much, much smaller). And for the storage of files, especially cold files, every percent improvement imparts a cost savings.
Also, widespread in-browser support of Brotli makes it possible for us to serve the dropbox files directly to browsers in the future (especially since they are concatenatable). Zstd browser support isn't at the same level today.
Re: Broccoli: Syncing Faster by Syncing Less
#28Is it possible to use this as rsync replacement ?
They aren't on the same level of abstraction. Rsync currently uses zlib for block compression on the wire. Brotli/broccoli would be an alternative option.
Re: Broccoli: Syncing Faster by Syncing Less
#29I wonder whether syncthing can use it.
Re: Broccoli: Syncing Faster by Syncing Less
#30This is why I continue to use Dropbox for daily work and constantly changing files. The syncing is unmatched. It’s surprising how bad the others like OneDrive and google drive are in comparison.