Live data from Hacker News

Dropbox’s Exodus from the Amazon Cloud

wired.com

191–200 of 252 posts

Re: Dropbox’s Exodus from the Amazon Cloud

#191
post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

What's it like working with Rust full-time? Is it similar to using any other language for a job? Do you find yourself enjoying coding more? Do you get unnecessarily caught up in micro-optimizations for things you wouldn't blink at in Go/Java?

Re: Dropbox’s Exodus from the Amazon Cloud

#192
post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

Which OS and FS the rust code is running on ?

No filesystem - the rust codebase directly handles the disk layout and scheduling.

Our previous version of the storage layer ran on top of XFS which was probably a bad choice since we ran into a few XFS bugs along the way, but nothing serious.

Re: Dropbox’s Exodus from the Amazon Cloud

#193
post #164

Earlier quoted context omitted.

Yes, balancing links and distributing work w/out congestion collapse became a really interesting and nontrivial problem.

I can't even imagine. Did you guys get into custom network hardware or still a Juniper/Cisco shop?

Juniper QFX from what I've heard.

Re: Dropbox’s Exodus from the Amazon Cloud

#194
post #131

I am curious to know, with so many disks densely packed in a 4U configuration, I am guessing there is definitely increased heat generation and not to mention vibration. How do you handle these? Also, does it have any effect on MTBF?

Yeah, our hardware teams put a lot of qualification time into hardware profiles before we buy in bulk. And part of what we test is power utilization, heat, etc, under normal software loads vs extreme loads, to make sure we don't exceed power budget, don't spin everything up at the same time, etc. This process can take months. A lot of the system design is about talking closely to vendors about MTBF for specific loads…

It is indeed scary to watch a storage node pull 1.5-2 kW at power-on. I think my eyes popped out the first time I watched the PDU load dance around like that.

Re: Dropbox’s Exodus from the Amazon Cloud

#195
post #163

Earlier quoted context omitted.

So NVMe is "ready for production"? Do you think you'd use it at home, or only in the enterprise?

I mean, if at home you need 400k IOPS, 1GB/s of writes, and 2.2GB/s of reads... go for it! I sure don't, but more power to you. :-)

NVMe's lower latency is great. I've never really known how "slow" regular SATA/SAS SSD's were until I used NVMe ones.

Re: Dropbox’s Exodus from the Amazon Cloud

#196
post #151

Earlier quoted context omitted.

> Are you using spindle or ssd or flash? (admittedly I dont know if you consider flash and ssd to be the same) We have flash caches (NVMe) in the machines, but the long-term storage devices are spindle--PMR and SMR. > What is your price per GB raw? I cannot disclose, exactly, but it is well below any other public price I've seen.

So NVMe is "ready for production"? Do you think you'd use it at home, or only in the enterprise?

I put it in production, and it's something else. Whether I'd use it at home depends entirely on how rich I feel I am ;)

Re: Dropbox’s Exodus from the Amazon Cloud

#198

Earlier quoted context omitted.

So NVMe is "ready for production"? Do you think you'd use it at home, or only in the enterprise?

We have been primarily using Intel NVMe storage for our database servers since fall of 2014 with no major complaints. Our high end desktop/laptop systems are also using the latest Samsung NVMe M.2 drives which are screaming fast.

Just to put a number to "screaming fast", 2.5 GB/s sequential reads averaged across the entire drive on a Samsung 950 Pro NVMe SSD: https://pbs.twimg.com/media/CaMhc2oVAAE4q3q.jpg:large

Re: Dropbox’s Exodus from the Amazon Cloud

#199
post #108
post #2

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

Do you know what Amazon is going to do with their excess 500PB of capacity? Are they scaling up fast enough that it isn't a big deal? Is Glacier selling your vacated storage? You must have left a large hole in AWS' revenue stream. I assume their fees for transferring your data out of their system helped cover that short-term, but 500PB of data is a lot of storage capacity to sell. Your other answers indicate your sep…

With storage you're constantly mothballing and replacing old iron behind the scenes, so Amazon might just ditch a bunch of old depreciated drives without hard feelings. AWS being on the up and up for SMEs, they are probably not crying over losing one large but very demanding customer -- and DB are still on AWS for European customers anyway.

Re: Dropbox’s Exodus from the Amazon Cloud

#200
post #84

Earlier quoted context omitted.

We have several dozen very, very good engineers working on a few discrete teams--infrastructure security, product security, etc. I don't feel comfortable getting any more specific than that; we keep security information pretty close to the vest.

To be honest, that response is less than comforting. But I understand how that'd be your answer given your position in the organization. Hopefully someone else at Dropbox feels like sharing more at some point and being more forthright about your company's practices in protecting data others entrust to them. Feel free to refer others to this thread if you think it'd be helpful.

That's disingenious. The awesome sec teams at AWS or Google are not going to review and maintain your internet-facing VM network config, which is where the real threats are. Sure, their firewalls might be a bit more robust (you don't really know, you can't see them), but anything behind them is just as insecure as you configure it yourself, and that's where penetration happens in real life. Threats at hw or kernel level in a hypervisor are extremely hard to exploit, by the time you get close to that stack you can usually see much easier and juicier targets to harvest.
Post reply on HN