Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox’s Exodus from the Amazon Cloud

wired.com

Post reply on HN
Print view View original

101–110 of 252 posts

Re: Dropbox’s Exodus from the Amazon Cloud

#101

Post by scott_s »

barkingcat wrote: post #72

The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

Dropbox is clearly a cloud company to its users. Whether or not they use a generic cloud, or maintain their own systems, is an implementation detail to users of the service.

Re: Dropbox’s Exodus from the Amazon Cloud

#105

Post by jamwt »

abhv wrote: post #102
jamwt wrote: post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

Does magic pocket use any non-standard error-correction algorithms, or just parity-style RAID5 or 6?

We use Reed Solomon variants with local reconstruction codes.

Re: Dropbox’s Exodus from the Amazon Cloud

#106

Post by jamwt »

nilayp wrote: post #100
james_cowling wrote: post #82

Earlier quoted context omitted.

We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.

Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.

We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.

Re: Dropbox’s Exodus from the Amazon Cloud

#107

Post by jamwt »

ju-st wrote: post #95
james_cowling wrote: post #49

Earlier quoted context omitted.

To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc). Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.

In datacenters space is cheap in comparison to power so often it's simply not economical to cram more in less space.

Sometimes; but depreciation costs are usually more than either of those (for storage).

Re: Dropbox’s Exodus from the Amazon Cloud

#108

Post by gvb »

jamwt wrote: post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

Do you know what Amazon is going to do with their excess 500PB of capacity? Are they scaling up fast enough that it isn't a big deal? Is Glacier selling your vacated storage?

You must have left a large hole in AWS' revenue stream. I assume their fees for transferring your data out of their system helped cover that short-term, but 500PB of data is a lot of storage capacity to sell.

Your other answers indicate your separation with AWS was very amicable, much more amicable than I would have expected. Either "its just business" or they have plans to backfill the hole you left.

Re: Dropbox’s Exodus from the Amazon Cloud

#109

Post by jabl »

jamwt wrote: post #25
dikaiosune wrote: post #17

Earlier quoted context omitted.

The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? If the latter, I've heard recent versions have had some significant improvements -- has your team looked at that and thought that you would have been OK if you just w…

Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the…

How do you do network io with rust? Thread-per-connection, non-blocking (using mio or?), or something else?

Re: Dropbox’s Exodus from the Amazon Cloud

#110

Post by jacquesm »

jamwt wrote: post #86

I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy t…

Very classy to be this transparent about what constitutes a team effort. The common narratives that focus on one or a very small number of visible people has a self-reinforcing effect where those key people get more name recognition and in turn that leads to them being even more visible the next time the team accomplishes something. It's rare for team leaders to step forward and to acknowledge the fact that it really was a team effort.
Post reply on HN