Live data from Hacker News

Broccoli: Syncing Faster by Syncing Less

dropbox.tech

11–20 of 58 posts

Re: Broccoli: Syncing Faster by Syncing Less

#12

In my opinion broccoli does not go so well with bread (brötli = bread roll in swiss german), so some more matching name suggestions are: gipfeli (Croissant), weggli, pfünderli (500g bread), bürli, zöpfli :-)

https://github.com/google/zopfli for the latter

Re: Broccoli: Syncing Faster by Syncing Less

#14

In my opinion broccoli does not go so well with bread (brötli = bread roll in swiss german), so some more matching name suggestions are: gipfeli (Croissant), weggli, pfünderli (500g bread), bürli, zöpfli :-)

Savory with a touch of sweetness, Broccoli Bread cooks up like cornbread but offers fiber and calcium. The original name was Brot-cat-li (since files could be concatenated and compressed in parallel), but when we said it fast it sounded like "Broccoli" and the name stuck.

Re: Broccoli: Syncing Faster by Syncing Less

#15

Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

Out of curiosity, how much does bandwidth usage contribute to your overall operational efficiency (as compared to for example the cost of running the actual servers)? Would totally understand if you can't answer this :)

Re: Broccoli: Syncing Faster by Syncing Less

#16

Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

Out of curiosity, how much does bandwidth usage contribute to your overall operational efficiency (as compared to for example the cost of running the actual servers)? Would totally understand if you can't answer this :)

Alexey from Traffic Team is here. Traffic is definitely a non-negligible part of the budget. We try to reduce it as much as possible for both lower operational expenses and better user experience. Main drivers for that improvement (besides owning our own Edge infrastructure) on the client side are:

1) Brotli (Broccoli) compression.

2) Differential updates through librsync.

3) "LANSync" a P2P sync within a broadcast domain (secured through server issued short lived TLS certs.)

That said, Desktop Client is only 1/3 of the overall Dropbox traffic -- the rest 2/3 are split between Web and API.

Re: Broccoli: Syncing Faster by Syncing Less

#17

Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

My understanding is Dropbox used to first hash file, then see if a copy was already uploaded. That was removed as it was being used for piracy.

Does Dropbox still upload everything, even if the user has uploaded it before?

Re: Broccoli: Syncing Faster by Syncing Less

#18

Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

My understanding is Dropbox used to first hash file, then see if a copy was already uploaded. That was removed as it was being used for piracy. Does Dropbox still upload everything, even if the user has uploaded it before?

if a user has uploaded a file in the past and their desktop client can prove they have access, then they can avoid uploading it again

Re: Broccoli: Syncing Faster by Syncing Less

#20
post #19

Curious if there's enough of any one type of file that a specialty compression for it would be worth the added complexity.

Great question! We developed and deployed Lepton to losslessly encode JPEG image files. Lepton continues to deliver substantial storage and cost savings every year. You can read more about it here https://dropbox.tech/infrastructure/lepton-image-compression...
Post reply on HN