Dropbox’s Exodus from the Amazon Cloud • Hacker News Board

Re: Dropbox’s Exodus from the Amazon Cloud

#71

Post by wpietri » Mon, Mar 14, 2016, 6:26 PM UTC

planetjones wrote: post #4

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the ch…

I think this is where your mistake comes in:

> I imagine a lot of people are like me i.e. Store most of the stuff at the cheapest location and use Dropbox just for docs that you want to sync on multiple devices

I don't believe that's a correct intuition. You're describing optimizer behavior, but for most things people are saticficers [1].

I happily pay Dropbox $100 a year to make a problem go away. Is it the best deal? I don't care. I'm not going to mess around over $0.27 per day. For me it's magic syncing and backup for all my important stuff. Using Dropbox may not be cash-optimal, as I'm currently paying something like $4 per GB/year. But for this I'm not an optimizer, I'm a satisficer, and as long as Dropbox maintains my experience of perfect reliability, I'll keep paying them.

[1] https://en.wikipedia.org/wiki/Satisficing

Re: Dropbox’s Exodus from the Amazon Cloud

#72

Post by barkingcat » Mon, Mar 14, 2016, 6:27 PM UTC

The word "cloud" loses all meaning in this article.

"The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company."

Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

Re: Dropbox’s Exodus from the Amazon Cloud

#73

Post by simonlc » Mon, Mar 14, 2016, 6:27 PM UTC

asendra wrote: post #22

So, just for fun. The video quotes "over 1PB" of storage per box. I count 6 columns of 15 rows/drives. Might be 7 columns even, there's some panels that aren't fully open on the video. So, 90-105 drives. I'm guessing they are using 10TB drives, although maybe they can get bigger unannounced drives? Roughly, the math seems to check out. Quite impressive. Guess the Backblaze guys need a Storage Pod 6.0 soon :P (I know,…

Someone said above that they support 10 and 14TB drives.

Re: Dropbox’s Exodus from the Amazon Cloud

#74

Post by illumin8 » Mon, Mar 14, 2016, 6:28 PM UTC

jamwt wrote: post #29

ignoramous wrote: post #24

Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Disko…

> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…

I had to count carefully, but it appears you're claiming 12 9's of durability for magic pocket, which seems like a subtle jab at S3's 11 9's.

I think both durability numbers would be fine for customers, but I've also wondered about the math behind AWS S3 durability for a while, and how you would prove statistically that 11 9's was even possible.

Re: Dropbox’s Exodus from the Amazon Cloud

#75

Post by true_religion » Mon, Mar 14, 2016, 6:31 PM UTC

iofj wrote: post #19

LeifCarrotson wrote: post #10

Earlier quoted context omitted.

And provided your management costs are lower than Amazon's profit on S3. There's an infection point somewhere at which owning your own hardware and paying someone to acquire, build, manage, and maintain it becomes cheaper than renting from someone who does the same thing at scale and makes a profit off of it. Dropbox is probably there, but many small web apps are not.

Keep in mind that that doesn't just result in more costs : it results in limits on scaling. Suppose you are on S3 and you can serve a customer x Mb/$. Now you're limited to products that you can charge more for. Suppose you get data served to customers on dedicated at x/10 Mb/$ (easily doable, and compared to S3 I bet x/50 would be doable). Isn't this how products like facebook video, vimeo (and thousands of porn sit…

I'd agree with that. Let me provide some numbers.

Currently, S3 will provide bandwidth out for 300TB/month at ~$21,000.

At our web host the same bandwidth would cost $1800.

At our CDN, the same bandwidth would cost $3585 to $4791 depending on the number of points of presence you'd require (6 vs 17).

S3 and its extreme availability profile is nice, but its not cost effective when the difference between pricing vs the lowest-priced option is almost $20,000 a month.

To me, the main alure of S3 was never amazon's architecture or HA, but that you no longer had to manage your storage servers and add capacity in a stepwise process. 7 years ago, to do this you were stuck with GlusterFS or even MongoFS.

Today we have OpenStack, Ceph, RiakCS and others providing battle-tested open-source solutions that anyone can run so there's much less of a reason to go S3.

I like to compare Amazon to Akamai with the statement of "yes, they're the best or close to it; but do we need the best?"

Re: Dropbox’s Exodus from the Amazon Cloud

#76

Post by wpietri » Mon, Mar 14, 2016, 6:31 PM UTC

planetjones wrote: post #4

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the ch…

By the way:

> downvoters - is HN unable to have a debate about whether this was a smart move or not

I doubt the downvotes are in opposition to discussing it. I think it's your poor framing of it. You basically say, "these guys are doomed because I imagine things". It's not other people's job to argue you out of your errors.

If you'd actually like to start a discussion, try saying something like, "I don't understand why X" or "How will Dropbox overcome factor Y". When you come out swinging, you start an oppositional dynamic, and downvotes are a common response to an aggressive opening that isn't so compellingly put that people feel it's worth responding to.

Re: Dropbox’s Exodus from the Amazon Cloud

#77

Post by djcapelis » Mon, Mar 14, 2016, 6:36 PM UTC

jamwt wrote: post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

Who's part of your security team over there? Most of the big cloud companies have people I trust working for them and I've been pretty comfortable telling most of my clients that cloud services are often going to have a better security team than they will. :)

But now that you've moved away from AWS, have you expanded your security team to help make up for the fact that you need folks to cover all your infrastructure security needs now too? Have you made the hires you needed to deal with the low level security issues in hw and kernel? How have you all been dealing with this so far?

I ask, because I help people make sense of their security issues and my clients explicitly ask me about your company. So keeping track of what you all are doing there is pretty relevant!

Re: Dropbox’s Exodus from the Amazon Cloud

#78

Post by atYevP » Mon, Mar 14, 2016, 6:37 PM UTC

james_cowling wrote: post #61

bluedino wrote: post #59

>> Transferring four petabytes of data, it turned out, took about a day. 46GB/s, is my math right?

We were pushing over a terabit of data transfer at peak, so 4-5PB per day.

Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!

Re: Dropbox’s Exodus from the Amazon Cloud

#79

Post by mandeepj » Mon, Mar 14, 2016, 6:41 PM UTC

james_cowling wrote: post #44

jamwt wrote: post #29

Earlier quoted context omitted.

> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…

Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon. We're always about the right tool for the job tho, and there are definitely use cases where Rust makes a lot of sense. We've been really happy with it so far.

Was not earlier it was Python? I guess Dropbox even hired Guido van Rossum due to this.

Re: Dropbox’s Exodus from the Amazon Cloud

#80

Post by james_cowling » Mon, Mar 14, 2016, 6:41 PM UTC

myth_drannon wrote: post #54

Now that Golang is making headways at Dropbox, I guess Python codebase will diminish in its importance... I wonder how Guido Van Rossum feels about it. First he was at Google but they created Go and he left to Dropbox which was a large Python shop, now they are moving to Go too.

I probably wasn't sufficiently clear in my other message re Golang:

- Python is our primary development language for most stuff at Dropbox.

- Mobile development and UI code use whatever is appropriate for each platform.

- Go is the primary language for Infrastructure, meaning fairly deep-backend stuff: databases, storage systems, message pipelines, search indices etc.

- Rust is used on Magic Pocket (which is within Infrastructure).

- We still have a bit of C++ code floating around but there's not much new development in C++ within Infrastructure. C/C++ are still used for common libraries for mobile development however.