Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox’s Exodus from the Amazon Cloud

wired.com

Post reply on HN
Print view View original

211–220 of 252 posts

Re: Dropbox’s Exodus from the Amazon Cloud

#211

Post by jl6 »

Sanddancer wrote: post #130
jl6 wrote: post #53

Earlier quoted context omitted.

I've never heard of a 14TB disk drive. Is that what you mean? Who makes these?

10 TB drives can be gotten off the shelf -- http://www.hgst.com/products/hard-drives/ultrastar-he10 -- and 14 TB drives are probably in the state of availability for large customers; it's not unusual for drive makers to make available drives coming down the pipe to certain customers.

True, it's just that they also tend to shout about upcoming drives and boast that they are previewing to select customers. Case in point, HGST marketing is in full swing on that He10 but good luck finding any stock in a mainstream retailer.

Re: Dropbox’s Exodus from the Amazon Cloud

#212

Post by panamafrank »

jamwt wrote: post #25
dikaiosune wrote: post #17

Earlier quoted context omitted.

The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? If the latter, I've heard recent versions have had some significant improvements -- has your team looked at that and thought that you would have been OK if you just w…

Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the…

Did you guys use a custom allocator for rust? And if so how did it differ from jemalloc and how could it be compared to C++ allocators like tbb::scalable_allocator?

Re: Dropbox’s Exodus from the Amazon Cloud

#213

Post by roel_v »

echelon wrote: post #171

Unrelated to the content of the article--I've never seen the oft talked about "anti-adblocker interstitial" before. I was surprised to find that the website blocks viewing of the article entirely because I'm running Adblock. It's an interesting subject. I'll never click on or be persuaded by ads, so they're not really gaining anything by showing me ads. I'm essentially worthless traffic to them no matter how they cut…

"I'll never click on"

Doesn't matter, impressions are valuable to marketers as well.

"or be persuaded by ads"

Yes you will, you just think you don't/won't.

Re: Dropbox’s Exodus from the Amazon Cloud

#214

Post by echelon »

roel_v wrote: post #213
echelon wrote: post #171

Unrelated to the content of the article--I've never seen the oft talked about "anti-adblocker interstitial" before. I was surprised to find that the website blocks viewing of the article entirely because I'm running Adblock. It's an interesting subject. I'll never click on or be persuaded by ads, so they're not really gaining anything by showing me ads. I'm essentially worthless traffic to them no matter how they cut…

"I'll never click on" Doesn't matter, impressions are valuable to marketers as well. "or be persuaded by ads" Yes you will, you just think you don't/won't.

> Yes you will, you just think you don't/won't.

That's a really blanket statement. I'm rather minimalist, and I typically form negative opinions of the things I see in ads.

Re: Dropbox’s Exodus from the Amazon Cloud

#215

Post by cm3 »

cm2187 wrote: post #210
james_cowling wrote: post #49

Earlier quoted context omitted.

To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc). Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.

Is dropbox using SMR? (I never used dropbox. Do they offer incremental backup? Otherwise I would have thought they would need to be able to do random writes, unless SMR is so cheap that it is economical to keep some dead data when a user uploads a new version of a file)

There was a post a while back here from a Dropbox employee explaining how they manage SMR disks manually via a direct HBM card and basically doing what the firmware would do otherwise. I can't find the post but it was talking about working around the architectural deficiencies of SMR while still getting to use more disk space.

Re: Dropbox’s Exodus from the Amazon Cloud

#216

Post by _urga »

jamwt wrote: post #52
james_cowling wrote: post #49

Earlier quoted context omitted.

To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc). Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.

Yes, a 1PB+ failure domain only made sense because Magic Pocket is very good at automatic cluster management and repair.

Surely it's the other way round?

That is, the better your software can handle repair, the smaller your failure domain can be?

A larger failure domain would only make sense if you wanted to minimize the compute required per unit stored?

Re: Dropbox’s Exodus from the Amazon Cloud

#217

Post by yread »

echelon wrote: post #171

Unrelated to the content of the article--I've never seen the oft talked about "anti-adblocker interstitial" before. I was surprised to find that the website blocks viewing of the article entirely because I'm running Adblock. It's an interesting subject. I'll never click on or be persuaded by ads, so they're not really gaining anything by showing me ads. I'm essentially worthless traffic to them no matter how they cut…

Yeah this stuff is getting annoying, my organization's firewall blocks ads as they consider them security risk so there's really nothing I can do about it. I guess I should just get back to work sigh

Re: Dropbox’s Exodus from the Amazon Cloud

#218

Post by kdkeyser »

james_cowling wrote: post #209
jaysoncena wrote: post #201

Earlier quoted context omitted.

this is really cool! Can you give more details on how blocks, replications, metadata, etc. are being handled?

Lazy answer but we'll blog about this in the next month. On a high level it's variable-sized blocks packed into 1GB extents, which are then aggregated into volumes and erasure coded across a set of disks on different machines/racks/rows/etc. We also replicate cross-country in addition to this. Live writes are written into non-erasure-coded volumes and encoded in the background. The volume metadata on the disks contai…

Are live writes replicated in real time, or are they locally staged (which is what Facebook does, I believe)

Also, do you mimic the eventually consistent behaviour of AWS, or do you offer a stronger form of consistency?

Re: Dropbox’s Exodus from the Amazon Cloud

#219

Post by deanCommie »

echelon wrote: post #214
roel_v wrote: post #213

Earlier quoted context omitted.

"I'll never click on" Doesn't matter, impressions are valuable to marketers as well. "or be persuaded by ads" Yes you will, you just think you don't/won't.

> Yes you will, you just think you don't/won't. That's a really blanket statement. I'm rather minimalist, and I typically form negative opinions of the things I see in ads.

You are one person. Statistics show most people are not like you.

Re: Dropbox’s Exodus from the Amazon Cloud

#220

Post by deanCommie »

jimbokun wrote: post #128
barkingcat wrote: post #72

The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

Was discussing "in the cloud" with my wife yesterday, and I said "It just means on some company's servers." Which suddenly de-mystified the whole concept for her.

I was at PDC in 2008 when Microsoft announced "Windows Azure".

I've never been more confused in my life. "Wait, what does this version of Windows do?"

Luckily they rebranded it as just "Microsoft Azure" which makes a shit ton more sense.

Post reply on HN