Pocket Watch: Verifying Exabytes of Data
blogs.dropbox.com
Pocket Watch: Verifying Exabytes of Data
1–10 of 28 posts
Re: Pocket Watch: Verifying Exabytes of Data
#2Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
Re: Pocket Watch: Verifying Exabytes of Data
#3Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing!
I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :)
Would Dropbox ever share hard drive reliability stats similar to what Backblaze does?
Sorry for all the questions in one comment! Not affiliated with either Dropbox or Backblaze, just an infrastructure wonk.
Re: Pocket Watch: Verifying Exabytes of Data
#4Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing! I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :) Would Dropbox ever share hard drive reliability sta…
Yeah to be clear my point from the article is that even though you can model an incredibly high durability number in theory, there's a lot more work required to see that in practice. I'm sure the S3 team also have extremely high internal durability numbers.
A lot of other factors go into an end-to-end durability numbers, including the possibility of software bugs, correlated failure, etc. Magic Pocket has an external durability claim of 12 9s which gives us a lot of headroom between the external figure and the internal model.
Fortunately both 11 or 12 nines is close enough to infinite as far as clients are concerned.
Re: Pocket Watch: Verifying Exabytes of Data
#5Earlier quoted context omitted.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing! I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :) Would Dropbox ever share hard drive reliability sta…
Yeah to be clear my point from the article is that even though you can model an incredibly high durability number in theory, there's a lot more work required to see that in practice. I'm sure the S3 team also have extremely high internal durability numbers. A lot of other factors go into an end-to-end durability numbers, including the possibility of software bugs, correlated failure, etc. Magic Pocket has an external…
Thanks for taking the time to reply James!
Re: Pocket Watch: Verifying Exabytes of Data
#6Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing! I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :) Would Dropbox ever share hard drive reliability sta…
27 9s is literally higher than my confidence that human civilization will be here in the next second. 10^27 seconds is about 32 quintillion years. Extinction events occur with a much higher frequency than that.
No criticism of Dropbox here; they know that number is just math games, too. I'm just putting some numbers on how true that is. Because I find tossing about big numbers like this as if they mean something a bizarre, nerdy sort of fun.
Re: Pocket Watch: Verifying Exabytes of Data
#7This system is pretty amazing. Thanks for sharing all the details as you went along.
You guys have made me interested in rust, which I still find a little too verbose vs golang. :D
Re: Pocket Watch: Verifying Exabytes of Data
#8Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing! I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :) Would Dropbox ever share hard drive reliability sta…
S3's durability numbers are for a single region (since they were quoting that number before they supported cross-region replication) -- so you'd get much better durability if you mirror your objects across regions. Which, apparently, is what Dropbox does.
Though if you require such high durability, you're better off with multiple providers in multiple countries, since at 11 9's, it probably becomes more likely that a provider will go out of business or a civil disturbance will make your data become unavailable than the chance of them losing it.
Re: Pocket Watch: Verifying Exabytes of Data
#9Earlier quoted context omitted.
Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s), while also charging less for storage and bandwidth than S3 would. Amazing! I would be curious to see how much of your verification architecture is shared with someone like Backblaze, but I assume the only way to learn that would be to work for both companies to compare :) Would Dropbox ever share hard drive reliability sta…
S3's durability numbers are for a single region (since they were quoting that number before they supported cross-region replication) -- so you'd get much better durability if you mirror your objects across regions. Which, apparently, is what Dropbox does. Though if you require such high durability, you're better off with multiple providers in multiple countries, since at 11 9's, it probably becomes more likely that a…
To reply your first paragraph, Dropbox is providing multi-region durability for free compared to S3. Point to Dropbox.
I agree with your second paragraph in entirety.
Re: Pocket Watch: Verifying Exabytes of Data
#10This system is pretty amazing. Thanks for sharing all the details as you went along. You guys have made me interested in rust, which I still find a little too verbose vs golang. :D
Thanks!
We're expanding our use of Rust into even more bold endeavours, details soon.
Based on our experience with it in the last 1.5 years, stick with it, it will return your effort many times over in the medium-long run. Then again, if there is no medium-long run for your project (as an entity needing maintenance), just use Python.