Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox’s Exodus from the Amazon Cloud

wired.com

Post reply on HN
Print view View original

91–100 of 252 posts

Re: Dropbox’s Exodus from the Amazon Cloud

#91

Post by james_cowling »

jamwt wrote: post #86

I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy t…

Couldn't agree with this more.

One of the things that's not immediately apparent in an initiative like this is the huge amount of effort required to provision, deploy and manage a system of this scale, across a whole bunch of different teams. We couldn't have done any of this without a tremendous group of SREs with the wisdom to invest heavily in automation, in addition to all the hard work.

We'll follow up in a week or two with a blog post highlighting some of the work here. I think the framework for disk remediation (automatically detecting failures, re-provisioning, etc) is particularly interesting.

Re: Dropbox’s Exodus from the Amazon Cloud

#92

Post by fivesigma »

barkingcat wrote: post #72

The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

Let us hope that the IoT buzzword doesn't get abused as much as "cloud" has in the past 4-5 years.

Re: Dropbox’s Exodus from the Amazon Cloud

#93

Post by timdoug »

james_cowling wrote: post #91
jamwt wrote: post #86

I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy t…

Couldn't agree with this more. One of the things that's not immediately apparent in an initiative like this is the huge amount of effort required to provision, deploy and manage a system of this scale, across a whole bunch of different teams. We couldn't have done any of this without a tremendous group of SREs with the wisdom to invest heavily in automation, in addition to all the hard work. We'll follow up in a week…

As the SRE writing the blog post on disk lifecycle, I can confirm it amongst others are in flight. :)

Re: Dropbox’s Exodus from the Amazon Cloud

#94

Post by jl6 »

benologist wrote: post #87
jl6 wrote: post #53

Earlier quoted context omitted.

I've never heard of a 14TB disk drive. Is that what you mean? Who makes these?

http://www.theregister.co.uk/2016/03/03/samsung_2_5_inch_15t... 2.5" 15TB ... and SSD, and expected to get a lot bigger too!

But that's not an SMR drive.

Re: Dropbox’s Exodus from the Amazon Cloud

#95

Post by ju-st »

james_cowling wrote: post #49
riobard wrote: post #37

> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here ( https://news.ycombinator.com/item?id=10541052…

To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc). Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.

In datacenters space is cheap in comparison to power so often it's simply not economical to cram more in less space.

Re: Dropbox’s Exodus from the Amazon Cloud

#96

Post by draw_down »

jamwt wrote: post #86

I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy t…

Very cool! Next challenge: shipping without the crunch time and late hours for the team :)

Re: Dropbox’s Exodus from the Amazon Cloud

#97

Post by azdle »

fivesigma wrote: post #92
barkingcat wrote: post #72

The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

Let us hope that the IoT buzzword doesn't get abused as much as "cloud" has in the past 4-5 years.

As someone who works for a company whose main business is IoT/M2M/Connected Devices/Embedded Stuff on the Internet, it already has been and it almost meaningless.

Re: Dropbox’s Exodus from the Amazon Cloud

#98

Post by draw_down »

planetjones wrote: post #4

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the ch…

It is a little weird in this context to argue that Amazon storage is cheaper unless you're at scale like Dropbox is. I mean, this is about Dropbox.

Re: Dropbox’s Exodus from the Amazon Cloud

#99

Post by virtuallynathan »

james_cowling wrote: post #61
bluedino wrote: post #59

>> Transferring four petabytes of data, it turned out, took about a day. 46GB/s, is my math right?

We were pushing over a terabit of data transfer at peak, so 4-5PB per day.

i.e. 1Tbps? Jeez! That is crazy.

Re: Dropbox’s Exodus from the Amazon Cloud

#100

Post by nilayp »

james_cowling wrote: post #82
atYevP wrote: post #78

Earlier quoted context omitted.

Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!

We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.

Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.
Post reply on HN