Dropbox saved $75M over two years by building its own infrastructure (2018) • Hacker News Board

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#81

Post by dswalter » Wed, Nov 18, 2020, 7:17 PM UTC

evrydayhustling wrote: post #74

ebiester wrote: post #6

Just as a reminder: You are not Dropbox. You personally need to run the numbers with both scenarios. Depending on your use case, it may be cheaper to be in AWS, and it may be cheaper to be co-located. If you are as big as Dropbox, it may be cheaper to build your own data center. It reminds me of the arguments for and against K8s. Most of the discussion is based on use case, and not considering the solution space for…

Here's another way that you're not Dropbox (unless you are): you are not a public company that gets points from investors for removing [$35M] dollars from COGS (cost of goods sold), even if you spend [$40M] on R&D + capitalized expenditures to replace it. Stories like this can be really misleading because large companies almost can't help but trick themselves into financial shenanigans. It's very possible that Dropbo…

Why must Amazon pass off its savings to your org, in this example, rather than taking most of those realized R&D savings to their own investors?

Sure, AWS may be constantly innovating on doing things more cheaply and scalably. Sometimes, those savings get passed off to the customer; but not most of it, and not necessarily all that frequently.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#82

Post by benhoyt » Wed, Nov 18, 2020, 7:18 PM UTC

llarsson wrote: post #57

Cloud is taking a taxi everywhere, bare metal is owning your own car. In one case, you pay a premium so that somebody else worries about all details and maintenance, and in the other, you take on that burden but don't pay the premium. This has always been true for everything where a service is provided to you. Great if your needs are small or not well known because there is no initial investment, but bad in the long…

Not a bad analogy, but the taxi also comes with a driver, which makes it expensive but allows you to get work done in the back while you're en route. It'd be like a cloud provider that came with a devops person dedicated to you.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#83

Post by PopeDotNinja » Wed, Nov 18, 2020, 7:18 PM UTC

ericbarrett wrote: post #61

calderarrow wrote: post #3

I think this quote really hits the nail on the head and confirms what a lot of people may have intuitively known about the value of cloud providers: But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. I think the m…

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

Basically it makes sense to set up your own infrastructure when your business IS your infrastructure.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#84

Post by toast0 » Wed, Nov 18, 2020, 7:20 PM UTC

shay_ker wrote: post #2

Does anyone know what the break-even point is for building/managing your own infrastructure, vs. using cloud services? I'm also curious - what's the rough time/cost of moving to your own infrastructure? I'm sure it depends on lots of things, but looking for a ballpark. 1 year + 10 engineers? 5 years + 100 engineers?

There's a bunch of steps along the way between AWS[1] and managing your own infrastructure.

Virtual private servers, bare metal hosting, cabinet, rack, or cage in a colo, and then operating your own datacenters.

The more you buy into the AWS services and not just EC2 instances, the harder it is to move out. In other parts of the thread, people are saying you need to have a good team to run your own infra, but you also need to have a good team to run on other people's infra, and debug their bugs without visibility, so that you can guide their techs to fixing your issues, so I don't think that using other people's services absolves you of having a good team.

Really the issue isn't age of company or number of engineers. It's the number of servers you need, and how stable that is. If you can't predict your server count 3 months out, you need to host with someone who has stock on hand to buffer your growth. If your server count is small, you get better geographic redundancy picking up an instance here and there from around the world from a single vendor; colo space is available everywhere, of course, but you would likely be dealing with different vendors in each locality.

If you can take advantage of growing and shrinking your deployment throughout the day in response to load, and there's a dramatic difference between peak and trough, it makes a lot of sense to be somewhere that you pay by the hour, instead of by the month or have to buy for the peak and let it idle.

[1] Or Google Cloud, or Azure, or Oracle Cloud or whoever.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#85

Post by birdyrooster » Wed, Nov 18, 2020, 7:20 PM UTC

ericbarrett wrote: post #61

calderarrow wrote: post #3

I think this quote really hits the nail on the head and confirms what a lot of people may have intuitively known about the value of cloud providers: But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. I think the m…

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

They aren’t shrinking at all, we have the same talent pool and are trying to thinly spread it over all of the problems we can solve with networked computers. The amount of problems you can solve with networked computers has scaled way faster than our industries capability to find talent and train employees.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#86

Post by heipei » Wed, Nov 18, 2020, 7:20 PM UTC

evrydayhustling wrote: post #74

ebiester wrote: post #6

Just as a reminder: You are not Dropbox. You personally need to run the numbers with both scenarios. Depending on your use case, it may be cheaper to be in AWS, and it may be cheaper to be co-located. If you are as big as Dropbox, it may be cheaper to build your own data center. It reminds me of the arguments for and against K8s. Most of the discussion is based on use case, and not considering the solution space for…

Here's another way that you're not Dropbox (unless you are): you are not a public company that gets points from investors for removing [$35M] dollars from COGS (cost of goods sold), even if you spend [$40M] on R&D + capitalized expenditures to replace it. Stories like this can be really misleading because large companies almost can't help but trick themselves into financial shenanigans. It's very possible that Dropbo…

I want to see the kind of R&D Amazon has to do to bring down their insane egress prices ;)

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#87

Post by aaronblohowiak » Wed, Nov 18, 2020, 7:20 PM UTC

Fumtumi wrote: post #44

aaronblohowiak wrote: post #30

Earlier quoted context omitted.

> Nonetheless, it also doesn't need to be all or nothing. You can easily combine a MultiCloud approach. There may be reasons to go multicloud but ease isn’t one of them. You double your infra support overhead (or more likely, half its quality) and have a “least common denominator” experience. The natural tendencies of large organizations is a diffusion of investment but the cheapest costs frequently come from a conce…

Bigger you are bigger the differences between teams and products and projects. You can leverage the high quality network infrastructure from Google while using your own DC for Compute Heavy Load. Use Azure for your Windows specific workloads. Go with AliCloud in China. You need to be big enough so that running it yourself is doable with a certain amount of quality. Which does imply many teams and workloads.

My employer does have a luxury of focus in its product offering, though we do have a moderately heterogeneous approach in development, certainly compared to many of the peers that operate at similar scale.

Heterogeneity in compute location has a multiplicative effect on accounting, security, capacity management, network management and is dilutive in terms of expertise -- instead of being able to justify the worlds leading experts in one system, you now need more staffing to cover a wider surface area (and they all need to have collaboration overhead to ensure they arent working at cross-purposes in strategy or tactic.)

I think this belief in marginal benefit from "right tool for the job" is a local-optimization where the costs of coordination and overhead are not borne locally and so are generally undervalued/discounted.

My employer runs on a single cloud provider, but -- do to its scale and closeness to core competency of our business -- we do operate our own CDN infrastructure, and this is a decision I happen to agree with. As a result of this division, I am acutely aware of the impact it can have on an engineering organization and only in certain specialized use-cases would advise considering DIY or multi-cloud.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#88

Post by walrus01 » Wed, Nov 18, 2020, 7:22 PM UTC

A good compromise halfway is to have your own equipment in colocation, at a facility where your IP transit upstreams are Depending on your scale might even be able to peer with them directly if you're big enough.

Or, if you are big enough to justify it you can order direct 10Gbps cross connects within a number of facilities to AWS.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#89

Post by Ericson2314 » Wed, Nov 18, 2020, 7:23 PM UTC

ericbarrett wrote: post #61

calderarrow wrote: post #3

I think this quote really hits the nail on the head and confirms what a lot of people may have intuitively known about the value of cloud providers: But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. I think the m…

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

But it's insane that running a for-higher data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) In a sane economy, data centers for higher would be a fully-commoditized barely-profitable common carrier with little natural monopoly.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#90

Post by richardwhiuk » Wed, Nov 18, 2020, 7:23 PM UTC

dswalter wrote: post #81

evrydayhustling wrote: post #74

Earlier quoted context omitted.

Here's another way that you're not Dropbox (unless you are): you are not a public company that gets points from investors for removing [$35M] dollars from COGS (cost of goods sold), even if you spend [$40M] on R&D + capitalized expenditures to replace it. Stories like this can be really misleading because large companies almost can't help but trick themselves into financial shenanigans. It's very possible that Dropbo…

Why must Amazon pass off its savings to your org, in this example, rather than taking most of those realized R&D savings to their own investors? Sure, AWS may be constantly innovating on doing things more cheaply and scalably. Sometimes, those savings get passed off to the customer; but not most of it, and not necessarily all that frequently.

Because Google and Microsoft are competing against them.