>> Transferring four petabytes of data, it turned out, took about a day. 46GB/s, is my math right?
We were pushing over a terabit of data transfer at peak, so 4-5PB per day.
Dropbox’s Exodus from the Amazon Cloud
61–70 of 252 posts
Re: Dropbox’s Exodus from the Amazon Cloud
#62Earlier quoted context omitted.
> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…
Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon. We're always about the right tool for the job tho, and there are definitely use cases where Rust makes a lot of sense. We've been really happy with it so far.
What happened to Python?
Re: Dropbox’s Exodus from the Amazon Cloud
#63Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Disko…
At the end of the blog post, they mention that they're working on creating the ability for European companies to store data in Germany if requested. This is interesting as the implication is that European businesses don't see the US as a safe place to store their data anymore.
It's not about safety, it's about EU privacy laws and Safe Harbor. It is desirable to store the data in-country to avoid the hassle of requiring/finding a provider with an EU approved Data Protection Agreement.
Re: Dropbox’s Exodus from the Amazon Cloud
#64So, just for fun. The video quotes "over 1PB" of storage per box. I count 6 columns of 15 rows/drives. Might be 7 columns even, there's some panels that aren't fully open on the video. So, 90-105 drives. I'm guessing they are using 10TB drives, although maybe they can get bigger unannounced drives? Roughly, the math seems to check out. Quite impressive. Guess the Backblaze guys need a Storage Pod 6.0 soon :P (I know,…
Yev from Backblaze here ->
Yea :D Though if we used 10TB drives we could get about 450TB in one pod, it would just be hilariously expensive for us to do so. Granted that's not 1PB per pod, but we'd imagine our costs are lower. But yes, different use-cases, and we're working on 6.0 ;-)
Re: Dropbox’s Exodus from the Amazon Cloud
#65> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here ( https://news.ycombinator.com/item?id=10541052…
Yev from Backblaze here -> Yup! It's pretty entertaining! Granted we only store 180TB b/c we use primarily 4TB drives and it's inexpensive for us to do so. If we had 10TB drives in there we'd be pushing up to 450TB per pod, but the price for us would increase dramatically. Pod 6.0 will be a bit more dense!
Re: Dropbox’s Exodus from the Amazon Cloud
#66Earlier quoted context omitted.
Please do a technical blog post on some of these: 1. Go vs Rust at Dropbox's scale and requirements. 2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket. 3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc) 4. Improvements in perf, stability, security, and cost. Thanks.
Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :) Will most likely start with the following: 1. Overall architecture and internals. 2. Verification and projection mechanisms, how we keep data safe. 3. How we manage a large amount of hardware (hundreds of thousands of disks) with a small te…
Yev here -> Let me just say you're welcome and thanks for joining the party :D
Re: Dropbox’s Exodus from the Amazon Cloud
#67Earlier quoted context omitted.
> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performa…
I'd imagine that if Backblaze is realizing great cost savings building their own storage pods from commodity off the shelf components, Dropbox's costs we're even lower per GB. Well done.
Yev here -> One of the reasons we started opening up about our Backblaze Pods and Hard Drive stats was so others would do so. We approve this message :D
Re: Dropbox’s Exodus from the Amazon Cloud
#68Earlier quoted context omitted.
Please do a technical blog post on some of these: 1. Go vs Rust at Dropbox's scale and requirements. 2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket. 3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc) 4. Improvements in perf, stability, security, and cost. Thanks.
Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :) Will most likely start with the following: 1. Overall architecture and internals. 2. Verification and projection mechanisms, how we keep data safe. 3. How we manage a large amount of hardware (hundreds of thousands of disks) with a small te…
I would be especially interested to hear how a focus on actual customer experience was maintained during all of this deep technology work.
One of the ways tech companies wrong is by putting the tech first and the users second. I think Dropbox's early competitive advantage was putting the user experience first. Everybody else had weird, complex systems; you folks just had a magic folder that was always correct. User focus like that is easier to pull off when it's just a few people. But once you break the customer problem up into enough pieces that hundreds of engineers can get involved, sometimes the tail starts wagging the dog and technically-oriented decisions start harming the user experience.
How did you folks avoid that?
Re: Dropbox’s Exodus from the Amazon Cloud
#69Earlier quoted context omitted.
Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon. We're always about the right tool for the job tho, and there are definitely use cases where Rust makes a lot of sense. We've been really happy with it so far.
What happened to Python?
We still have a ton of Python. James is referring to the infrastructure software layer--storage services, monitoring, etc. That particular part of Dropbox is mostly Go.
But our web controller code, for example, is millions of lines of Python. And we use Python in lots of other places as well.
Re: Dropbox’s Exodus from the Amazon Cloud
#70Earlier quoted context omitted.
Please do a technical blog post on some of these: 1. Go vs Rust at Dropbox's scale and requirements. 2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket. 3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc) 4. Improvements in perf, stability, security, and cost. Thanks.
Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :) Will most likely start with the following: 1. Overall architecture and internals. 2. Verification and projection mechanisms, how we keep data safe. 3. How we manage a large amount of hardware (hundreds of thousands of disks) with a small te…
> Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :)
Here's a request, how did you migrate all that data. Are you going to be open-sourcing any of the tools that you built up? We'd be interesting in hearing about that process. A lot of folks using B2 right now are trying to do this exact thing (or making copies if not actual migrations). Cheers! ;-)