Dropbox’s Exodus from the Amazon Cloud • Hacker News Board

Re: Dropbox’s Exodus from the Amazon Cloud

#41

Post by jamwt » Mon, Mar 14, 2016, 5:12 PM UTC

JonFish85 wrote: post #30

jbob2000 wrote: post #9

Earlier quoted context omitted.

Their service is primarily for consumers and many large companies won't use them, favoring an in-house solution or an offering from a specialized company. That's a dangerous position to be in, as consumers are fickle and are liable to drop the service at a moment's notice. Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of…

"Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of people. I'm sure we'll hear about some new paid services from them which will be compliant with all manner of regulations. This will open them up to huge multi-million dollar contracts and they'll get baked into some big companies for years to come." Isn't that Box's busi…

Little known fact, but already we have more business customers than Box:

https://www.dropbox.com/business/customers

Re: Dropbox’s Exodus from the Amazon Cloud

#42

Post by toyg » Mon, Mar 14, 2016, 5:15 PM UTC

jbob2000 wrote: post #9

planetjones wrote: post #4

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the ch…

Their service is primarily for consumers and many large companies won't use them, favoring an in-house solution or an offering from a specialized company. That's a dangerous position to be in, as consumers are fickle and are liable to drop the service at a moment's notice. Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of…

> Dropbox is probably trying to position themselves as a secure data storage solution for larger organizations. Being on Amazon scares away a lot of people.

I don't think the people scared by Amazon would not be scared by a company with Condoleeza Rice on the board. Amazon is already compliant with all those regulations you speak of -- they host large swaths of USGov infrastructure, after all.

More likely, DB people have simply come to the realization that, at their scale, AWS is extremely expensive for what it delivers. Being a technology company, they are not scared by the thought of building and maintaining their own iron in order to make substantial savings. Their core business is storage, after all.

Re: Dropbox’s Exodus from the Amazon Cloud

#43

Post by james_cowling » Mon, Mar 14, 2016, 5:17 PM UTC

jedisct1 wrote: post #34

jamwt wrote: post #25

Earlier quoted context omitted.

Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the…

Are you guys hiring Rust developers by chance? Asking for a friend :)

Very possibly :) Drop me an email at james@dropbox.com.

Re: Dropbox’s Exodus from the Amazon Cloud

#44

Post by james_cowling » Mon, Mar 14, 2016, 5:20 PM UTC

jamwt wrote: post #29

ignoramous wrote: post #24

Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Disko…

> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…

Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon.

We're always about the right tool for the job tho, and there are definitely use cases where Rust makes a lot of sense. We've been really happy with it so far.

Re: Dropbox’s Exodus from the Amazon Cloud

#45

Post by jamwt » Mon, Mar 14, 2016, 5:22 PM UTC

anp wrote: post #28

jamwt wrote: post #25

Earlier quoted context omitted.

Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the…

Thanks very much for the detailed and thoughtful answers! I've read before (somewhere, I think) that Dropbox effectively maintains a large internal "standard library" rather than relying on external open source efforts. How much does Magic Pocket rely on Rust's standard library and the crates.io ecosystem? Could you elaborate on how you ended up going in whichever direction you chose with regards to third-party open…

We use 3rd parties for the "obvious" stuff. Like, we're not going to reinvent json serialization. But we typically don't use any 3rd party frameworks on the backend. So things like service management/discovery, rpc, error handling, monitoring, metadata storage, etc etc, are a big in-house stack.

So, we use quite a few crates for the things it makes no sense to specialize in Dropbox-specific ways.

Re: Dropbox’s Exodus from the Amazon Cloud

#46

Post by james_cowling » Mon, Mar 14, 2016, 5:28 PM UTC

ignoramous wrote: post #24

Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Disko…

This is most accurate. A couple of comments:

> Dropbox still uses AWS for its European customers.

We haven't publicly launched EU storage yet but will be doing so later in the year.

> Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally)

Facebook and Google and startup folks and people from random other places.

Our IOPS demands are reasonably modest on a per-disk basis which is why we skew heavily towards storage density. This is a different workload than you'd find at some of the other big players, hence a fairly custom solution.

> Not a political move

Definitely. AWS are great, we love working with them and will continue to do so.

Re: Dropbox’s Exodus from the Amazon Cloud

#47

Post by SeanDav » Mon, Mar 14, 2016, 5:29 PM UTC

james_cowling wrote: post #44

jamwt wrote: post #29

Earlier quoted context omitted.

> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…

Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon. We're always about the right tool for the job tho, and there are definitely use cases where Rust makes a lot of sense. We've been really happy with it so far.

Without getting into any sort of religious war, what are typical use cases that make Rust a better choice?

Re: Dropbox’s Exodus from the Amazon Cloud

#48

Post by dikaiosune » Mon, Mar 14, 2016, 5:32 PM UTC

jamwt wrote: post #45

anp wrote: post #28

Earlier quoted context omitted.

Thanks very much for the detailed and thoughtful answers! I've read before (somewhere, I think) that Dropbox effectively maintains a large internal "standard library" rather than relying on external open source efforts. How much does Magic Pocket rely on Rust's standard library and the crates.io ecosystem? Could you elaborate on how you ended up going in whichever direction you chose with regards to third-party open…

We use 3rd parties for the "obvious" stuff. Like, we're not going to reinvent json serialization. But we typically don't use any 3rd party frameworks on the backend. So things like service management/discovery, rpc, error handling, monitoring, metadata storage, etc etc, are a big in-house stack. So, we use quite a few crates for the things it makes no sense to specialize in Dropbox-specific ways.

Cool. This might be getting into the weeds a bit, but are you still on rustc-serialize for json or are you trying to keep up with serde/serde_json? If you're using serde, are you on nightly? From your comment above I got the impression that only using stable features was very important, so I'm curious how your codebase implements/derives the serde traits.

Re: Dropbox’s Exodus from the Amazon Cloud

#49

Post by james_cowling » Mon, Mar 14, 2016, 5:33 PM UTC

riobard wrote: post #37

> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here ( https://news.ycombinator.com/item?id=10541052…

To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc).

Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.

Re: Dropbox’s Exodus from the Amazon Cloud

#50

Post by skuhn » Mon, Mar 14, 2016, 5:34 PM UTC

gshx wrote: post #7

planetjones wrote: post #4

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the ch…

Not at all, cost/TB of S3 is much higher compared to what you can build provided you know how to build it yourself.

Having done the math on this project in a previous life, I would also add that having to run app logic on EC2 instances and the outbound network costs are the true financial killers for S3.

S3 raw price-per-gig is somewhat competitive with what Dropbox built, but the ancillary things around it are not remotely competitive.