Dropbox’s Exodus from the Amazon Cloud

Post by toomuchtodo » Mon, Mar 14, 2016, 4:45 PM UTC

jamwt wrote: post #29

ignoramous wrote: post #24

Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Disko…

> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…

Post by jedisct1 » Mon, Mar 14, 2016, 4:56 PM UTC

jamwt wrote: post #25

dikaiosune wrote: post #17

Earlier quoted context omitted.

The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? If the latter, I've heard recent versions have had some significant improvements -- has your team looked at that and thought that you would have been OK if you just w…

Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the…

Post by ethanbond » Mon, Mar 14, 2016, 5:06 PM UTC

JonFish85 wrote: post #30

jbob2000 wrote: post #9

Earlier quoted context omitted.

Their service is primarily for consumers and many large companies won't use them, favoring an in-house solution or an offering from a specialized company. That's a dangerous position to be in, as consumers are fickle and are liable to drop the service at a moment's notice. Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of…

"Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of people. I'm sure we'll hear about some new paid services from them which will be compliant with all manner of regulations. This will open them up to huge multi-million dollar contracts and they'll get baked into some big companies for years to come." Isn't that Box's busi…

Post by jamwt » Mon, Mar 14, 2016, 5:07 PM UTC

tedmielczarek wrote: post #35

jamwt wrote: post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

I'm curious what your inter-language interop situation looks like. Do you have any Go code calling directly into Rust code (or vice versa)? Are they segregated into standalone programs? If so, are they doing communication via some form of IPC? Cool stuff, really interesting read!

Post by riobard » Mon, Mar 14, 2016, 5:11 PM UTC

jamwt wrote: post #39

riobard wrote: post #37

> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here ( https://news.ycombinator.com/item?id=10541052…

The difference to get even higher is our system supports host-managed SMR disks that come in 10T and 14T sizes.