Dropbox saved $75M over two years by building its own infrastructure (2018) • Hacker News Board

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#161

Post by aaronblohowiak » Wed, Nov 18, 2020, 9:18 PM UTC

Fumtumi wrote: post #106

aaronblohowiak wrote: post #87

Earlier quoted context omitted.

My employer does have a luxury of focus in its product offering, though we do have a moderately heterogeneous approach in development, certainly compared to many of the peers that operate at similar scale. Heterogeneity in compute location has a multiplicative effect on accounting, security, capacity management, network management and is dilutive in terms of expertise -- instead of being able to justify the worlds le…

You also need to be on MultiCloud if you do not operate stuff on yourself so you are in a better negotiation position. Or so that you are not dependend on only one.

I hear this sentiment repeated frequently, but I’ve never heard multicloud as leverage actually getting a better deal than an exclusivity deal. If you have a different experience I’d love to connect and learn more - email in my profile.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#162

Post by oblio » Wed, Nov 18, 2020, 9:19 PM UTC

heipei wrote: post #86

evrydayhustling wrote: post #74

Earlier quoted context omitted.

Here's another way that you're not Dropbox (unless you are): you are not a public company that gets points from investors for removing [$35M] dollars from COGS (cost of goods sold), even if you spend [$40M] on R&D + capitalized expenditures to replace it. Stories like this can be really misleading because large companies almost can't help but trick themselves into financial shenanigans. It's very possible that Dropbo…

I want to see the kind of R&D Amazon has to do to bring down their insane egress prices ;)

They probably don't want the high egress business, anyway. Besides the whole "keep data in". They have a CDN for those businesses that need it and everything else that needs a lot of egress, they probably don't want as business.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#163

Post by Daishiman » Wed, Nov 18, 2020, 9:20 PM UTC

acoard wrote: post #151

marta_morena_28 wrote: post #138

Earlier quoted context omitted.

> 75 million dollar price tag is also huge. Is it? Over the two years they saved, this is about the salary for 100 engineers. Can you replace and maintain all the cloud aspects that AWS provides you with (I mean the ones you actually use) with 100 engineers? Maybe, if they are good engineers (which is kinda implied by the 300k salary tag in the calculation). If it's worth it, remains to be seen. Definitely nothing fo…

>Definitely nothing for any medium sized company. I work at a medium sized company. Depends on who you count, but let's say around ~30 devs. Recently we basically did just this, and it's been a great success. We haven't fully migrated and still use AWS for prod, but have seen substantial savings already. We spent $2k on servers, Dell r720s. We bought a UPS and mount, and racked them in our office. I installed OpenShi…

This post proves parent's point though.

You're not doing anything even remotely close to the features offered by cloud providers or even managed hosting providers.

Disaster recovery? Geographically separate redundant servers with failovers? Automated (and proven to work) backups? One-stop access control for infra maintenance? Audit controls for your database and storage objects? Tape backups?

Even today to support all those things you need a small army of specialists. Granted, a heck of a lot of things can get away with not having any of this. But the use cases are out there and hosting and maintaining all of that in-prem is another different level.

I understand your use case, but your is very, very far from the sheer and absolute complexity and features that enterprise data centers have.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#164

Post by blackaspen » Wed, Nov 18, 2020, 9:20 PM UTC

thor24 wrote: post #118

spullara wrote: post #103

Earlier quoted context omitted.

There are now businesses that explicitly depend on the elasticity of the cloud and can never really be moved on premise without massive up-front investment in hardware that may only be used a few times a year for their biggest customers. Trying to hybridize these workloads hasn't been very successful as of yet. It is possible that K8S could relive this problem but I haven't seen it in practice, at scale.

Instant Elasticity in Cloud is a myth. If you think you are going to get 1k hosts just like that from AWS you will have an unpleasant experience. I work at one of the decent size tech company and we are split between cloud and on prem. From our experience you have to inform AWS/GCP in advance (sometime way early) if you are looking to meaningfully increase capacity in zone/region. Sure, auto scaling few hundreds of h…

This is the same with disaster recovery too. The idea that "oh, our main DC went down, we'll just spin it up in another region" is great until you realize that means you need reserved instances in another zone, that just like another physical DC, you won't be using.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#165

Post by blackaspen » Wed, Nov 18, 2020, 9:23 PM UTC

betaby wrote: post #80

ericbarrett wrote: post #61

Earlier quoted context omitted.

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

There is a lot of spectrum in the middle between AWS micro and your own datacenter(s). One don't have to jump from the cloud to datacenter - just rent/buy a server or two, or rack of server or two.

Sure, but the management overhead from 1 server to 20 racks scales, at best, linearly. I've seen a lot of places just past the point where a single sys-ad person with a thumb drive is viable, yet they choose to not invest here and instead spend 2-5x on going to the cloud.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#166

Post by Daishiman » Wed, Nov 18, 2020, 9:23 PM UTC

cbdumas wrote: post #109

hengheng wrote: post #100

Earlier quoted context omitted.

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of…

IMO your assertion is validated by the excellent overview of Stack Overflow's infrastructure given here: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... Very few web apps will ever serve as much traffic as SO.

SO doesn't have a very operationally complex app.

A bank running 50 different services, on different platforms, with serious audit requirements, physical and logical access control, strict change and configuration management, etc., has two orders of magnitude more complexity. And that shit is very expensive in manpower.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#167

Post by easton » Wed, Nov 18, 2020, 9:24 PM UTC

thor24 wrote: post #118

spullara wrote: post #103

Earlier quoted context omitted.

There are now businesses that explicitly depend on the elasticity of the cloud and can never really be moved on premise without massive up-front investment in hardware that may only be used a few times a year for their biggest customers. Trying to hybridize these workloads hasn't been very successful as of yet. It is possible that K8S could relive this problem but I haven't seen it in practice, at scale.

Instant Elasticity in Cloud is a myth. If you think you are going to get 1k hosts just like that from AWS you will have an unpleasant experience. I work at one of the decent size tech company and we are split between cloud and on prem. From our experience you have to inform AWS/GCP in advance (sometime way early) if you are looking to meaningfully increase capacity in zone/region. Sure, auto scaling few hundreds of h…

How early is "way early"? Because as long as it's shorter than the two-three weeks it'd take to order boxes, rack them, provision them (which would be automated but might still take a afternoon), deal with any QA hiccups... I'd much rather call my AWS rep and say "can we add 30% by Thursday" and have them figure it out (and at such a large scale you might be able to spread it out across a couple regions anyway unless you only serve a specific part of the world).

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#168

Post by freeone3000 » Wed, Nov 18, 2020, 9:25 PM UTC

echelon wrote: post #117

dodobirdlord wrote: post #97

Earlier quoted context omitted.

Reliability concerns make datacenters resilient to commoditization. A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. Commodity businesses are largely built on presumptions of linearity. Produce 90% as much corn/iron/wood/widgets as you expected and you’ll probably make something like 90% of the money you expected. Produce a 90% available datac…

Then you should be able to sue your provider for breaches in SLA. 0.5% downtime should void the bill. I think presently the providers are getting all the upside. High margin, perpetual lock-in, and no consequences.

You can sue your provider for breaches in SLA. The rebate you automatically get likely exceeds what you'd otherwise recover. (Which is why they grant one). SLA payouts are usually very generous - I've gotten credit for the entire month for a one-hour outage of a service.

I don't know where this idea that cloud providers and other DCs don't pay out for SLA agreements is, but they absolutely do.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#169

Post by Daishiman » Wed, Nov 18, 2020, 9:25 PM UTC

dnautics wrote: post #137

hengheng wrote: post #100

Earlier quoted context omitted.

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of…

You still want ha, failover, and disaster recovery. Then you need to set up stuff like bgp, dns, security rules, etc, etc etc. Complexity mounts pretty quickly.

Indeed. It seems that most of the people saying that cloud hosting is expensive have never run into the issues of making their own SAN, managing the provisioning of 20 different teams, etc.

The organizational complexity and specialist knowledge is mind-boggling and there is zero chance that your in-house knowledge is better than what Amazon can provide.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#170

Post by Daishiman » Wed, Nov 18, 2020, 9:29 PM UTC

fakedang wrote: post #135

jasode wrote: post #115

Earlier quoted context omitted.

>But it's insane that running a for-higher[sic] data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) Your analysis is incomplete which is why it looks like insane high-profit margins. Amazon AWS (and MS Azure, Google Cloud) also sell high-value services on top of raw datacenters. It's not just commodity rack servers. Amazon keeps iterating…

Recently I just began using cloud services. I started with Firebase, thought it was super cool, then moved to GCP, which was super inflexible (and super shit support). So I decided to try Azure, since we used that at a previous startup, but it was too complicated to even get started. So I decided to give Jeff my money (or rather avail of his free tier), and started using AWS. While it's super complicated to use compa…

> The only solution to break out of such a duopoly would be for tech oriented companies to stop being lazy and start building out their own cloud infra.

What competitive advantage do you get by making your own in-house, inferior version of an infrastructure service that won't benefit from AWSs economies of scale?