Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Broccoli: Syncing Faster by Syncing Less

dropbox.tech

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Broccoli: Syncing Faster by Syncing Less

#41

Post by daniel_rh »

telendram wrote: post #38
daniel_rh wrote: post #27

Earlier quoted context omitted.

We heavily investigated zstd and met with the brilliant inventor, Yann, who provided amazing insights into the design and rationale behind zstd and why it is so fast and such an amazing technology. I also recompiled zstd into rust using https://github.com/immunant/c2rust and tried using various webasm mechanisms to run it (I didn't get webasm quite fast enough, and teaching c2rust to make it safe would be quite a slo…

> the main reason we settled on Brotli was the second order context modeling This advanced feature is only relevant on reaching compression levels 10 or 11, which are extremely slow. Below that, it's barely used by the encoder, due to memory and cpu taxes. Given your application has reached speed concerns, and ends up using brotli at compression level 1 in production, you would be surprised to notice that in this spe…

For long term storage of blocks, we compress at much higher compression levels like you mention. These densely compressed blocks are, in turn, served directly to customers when they download their own files.

For uploads you're right: we'd be theoretically better off with high performing zstd, but there are maintenance costs with maintaining 2 separate compression pipelines that are similar, but different, for upload and downloads.

Plus there is no safe rust zstd compressor and the safe rust zstd decompressor linked in this thread is only recently available and is also several times slower than the safe rust brotli decompressor.

Re: Broccoli: Syncing Faster by Syncing Less

#42

Post by glandium »

rspoerri wrote: post #11

In my opinion broccoli does not go so well with bread (brötli = bread roll in swiss german), so some more matching name suggestions are: gipfeli (Croissant), weggli, pfünderli (500g bread), bürli, zöpfli :-)

But it goes well with courgette. https://www.chromium.org/developers/design-documents/softwar...

Re: Broccoli: Syncing Faster by Syncing Less

#43

Post by ghj »

SaveTheRbtz wrote: post #16
adamsvystun wrote: post #15

Earlier quoted context omitted.

Out of curiosity, how much does bandwidth usage contribute to your overall operational efficiency (as compared to for example the cost of running the actual servers)? Would totally understand if you can't answer this :)

Alexey from Traffic Team is here. Traffic is definitely a non-negligible part of the budget. We try to reduce it as much as possible for both lower operational expenses and better user experience. Main drivers for that improvement (besides owning our own Edge infrastructure) on the client side are: 1) Brotli (Broccoli) compression. 2) Differential updates through librsync. 3) "LANSync" a P2P sync within a broadcast d…

> Desktop Client is only 1/3 of the overall Dropbox traffic -- the rest 2/3 are split between Web and API

Interesting! I assume the desktop client is still dropbox's main product so that's surprising to hear. Is it because the desktop has everything cached and rarely has to download whereas web and mobile has to download a fresh copy each time they are viewed?

Re: Broccoli: Syncing Faster by Syncing Less

#44

Post by datanut »

SaveTheRbtz wrote: post #16
adamsvystun wrote: post #15

Earlier quoted context omitted.

Out of curiosity, how much does bandwidth usage contribute to your overall operational efficiency (as compared to for example the cost of running the actual servers)? Would totally understand if you can't answer this :)

Alexey from Traffic Team is here. Traffic is definitely a non-negligible part of the budget. We try to reduce it as much as possible for both lower operational expenses and better user experience. Main drivers for that improvement (besides owning our own Edge infrastructure) on the client side are: 1) Brotli (Broccoli) compression. 2) Differential updates through librsync. 3) "LANSync" a P2P sync within a broadcast d…

> the rest 2/3 are split between Web and API.

Does this ratio include the Dropbox official mobile apps?

Have LANsync peers been considered as a sources of blocks for mobile clients?

Like most, I’m observing (and participating in) multidimensional access to data. For not, accessing files on my local desktop is still much faster than direct downloads from the Dropbox cloud. It’s a bummer to source files that are on my LAN from the cloud. This may become more problematic as bandwidth billing models move toward pay-per-bit.

Re: Broccoli: Syncing Faster by Syncing Less

#45

Post by jeffbee »

The tradeoff between client CPU time and upload speed is interesting. If they need to be able to output compressed text at 100mbps, that gives a budget of ~100ns/byte, or pretty much what they would have been spending with zlib in the first place. But on my fiber connection I only have a budget of 10ns/byte. Does that mean you'd use the equivalent of `brotli -q 1` for me? If so, doesn't the march of progress continually erode the advantages of compression in this use case?

Re: Broccoli: Syncing Faster by Syncing Less

#46

Post by lifthrasiir »

> Maintaining a static list of the most common incompressible types within Dropbox and doing constant time checks against it in order to decide if we want to compress blocks

There is also a format-agnostic and adaptable heuristic to stop compression if the initial part (say, first 1MB) of the file seems incompressible. I'm not sure whether this is widespread, but I've seen at least one software doing that and it worked well. This can be combined with other kinds of heuristics like entropy estimation.

Re: Broccoli: Syncing Faster by Syncing Less

#48

Post by nerdponx »

lanius wrote: post #47

Is there a pun between Broccoli and Brotli I'm not aware of? There's another Brotli compression tool called Broccoli (written in Go), just a coincidence?

We codenamed the Brotli compressor in Rust “Broccoli” because of the capability to make Brotli files concatenate with one another (brot-cat-li).

Re: Broccoli: Syncing Faster by Syncing Less

#49

Post by Hamuko »

manigandham wrote: post #30

This is why I continue to use Dropbox for daily work and constantly changing files. The syncing is unmatched. It’s surprising how bad the others like OneDrive and google drive are in comparison.

I recently switched from Dropbox because of the added device limitations for the free tier and because I don't really want to pay 10 euro a month for 2 TB of space when I only need 10 GB. Got myself a Nextcloud instance for third of the cost and I have to say that the syncing absolutely sucks. It's so bad that I'm going to migrate away from it as well.

Not going back to Dropbox yet though. I'd rather try out Google Drive since I consider it to be much better consumer plans.

Re: Broccoli: Syncing Faster by Syncing Less

#50

Post by yjftsjthsd-h »

CJefferson wrote: post #17
daniel_rh wrote: post #2

Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

My understanding is Dropbox used to first hash file, then see if a copy was already uploaded. That was removed as it was being used for piracy. Does Dropbox still upload everything, even if the user has uploaded it before?

> My understanding is Dropbox used to first hash file, then see if a copy was already uploaded. That was removed as it was being used for piracy.

How's that work? Somehow modify the client to say that you have a file with a user-provided hash even though it doesn't actually exist on disk?

Post reply on HN