Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox saved $75M over two years by building its own infrastructure (2018)

geekwire.com

Post reply on HN
Print view View original

131–140 of 218 posts

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#131

Post by acdha »

mikekoscinski wrote: post #101

Some assumptions to analyze this switch: - Dropbox saves $50M/year in AWS costs - Dropbox spends $200K/year (salary, benefits, equipment, SaaS, etc.) for their average infrastructure engineer Following those assumptions, Dropbox must hire Of course, these assumptions may be wrong (please correct them if so!) and this entirely ignores the unique computing needs of Dropbox's business, which may be unique vs. anything a…

> - Dropbox spends $200K/year (salary, benefits, equipment, SaaS, etc.) for their average infrastructure engineer

The usual rule of thumb is that you assume 2:1 for salary and support costs — that starts to drift at the higher end of the pay scale but I'd doubt that a company in San Francisco isn't paying a substantial amount for office space, health insurance, etc.

Your general point is still correct: call it a hundred engineers to break-even and it's likely still a substantial win, and there are some interesting angles for additional cost-optimization given Dropbox's mix of high network traffic and long-tail storage.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#132

Post by dec0dedab0de »

benhoyt wrote: post #82
llarsson wrote: post #57

Cloud is taking a taxi everywhere, bare metal is owning your own car. In one case, you pay a premium so that somebody else worries about all details and maintenance, and in the other, you take on that burden but don't pay the premium. This has always been true for everything where a service is provided to you. Great if your needs are small or not well known because there is no initial investment, but bad in the long…

Not a bad analogy, but the taxi also comes with a driver, which makes it expensive but allows you to get work done in the back while you're en route. It'd be like a cloud provider that came with a devops person dedicated to you.

Sure, but if you are using the taxi often enough it may be cheaper to higher a full time driver.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#133

Post by walrus01 »

cbg0 wrote: post #127
walrus01 wrote: post #88

A good compromise halfway is to have your own equipment in colocation, at a facility where your IP transit upstreams are Depending on your scale might even be able to peer with them directly if you're big enough. Or, if you are big enough to justify it you can order direct 10Gbps cross connects within a number of facilities to AWS.

> So, essentially, in the same building or across the street? May be very tricky to pull off.

Not really difficult, colocation is available in the same buildings as the largest IX points... Places like One Wilshire, 350 E. Cermak, NAP of the Americas, 60 Hudson, etc. It definitely has a premium price.

If you can extend your horizon of view out to buildings that are in the same metro area but less than 10, 15 or 20 km of (OTDR measured) fiber from the IX point, the performance is very nearly the same, and there's a lot more options. One example would be the datacenters in Tukwila, WA and their relationship to IX points in downtown Seattle.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#134

Post by vp8989 »

cbdumas wrote: post #109
hengheng wrote: post #100

Earlier quoted context omitted.

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of…

IMO your assertion is validated by the excellent overview of Stack Overflow's infrastructure given here: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... Very few web apps will ever serve as much traffic as SO.

"Very few web apps will ever serve as much traffic as SO."

Their traffic is like 80-90% reads and they actually hire good devs and let them work on perf.

Neither of those things are true in typical companies.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#135

Post by fakedang »

jasode wrote: post #115
Ericson2314 wrote: post #89

Earlier quoted context omitted.

But it's insane that running a for-higher data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) In a sane economy, data centers for higher would be a fully-commoditized barely-profitable common carrier with little natural monopoly.

>But it's insane that running a for-higher[sic] data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) Your analysis is incomplete which is why it looks like insane high-profit margins. Amazon AWS (and MS Azure, Google Cloud) also sell high-value services on top of raw datacenters. It's not just commodity rack servers. Amazon keeps iterating…

Recently I just began using cloud services. I started with Firebase, thought it was super cool, then moved to GCP, which was super inflexible (and super shit support). So I decided to try Azure, since we used that at a previous startup, but it was too complicated to even get started. So I decided to give Jeff my money (or rather avail of his free tier), and started using AWS. While it's super complicated to use compared to Firebase, it was relatively easy to learn from scratch due to huge amount of support online and what they have on offer. But what honestly stumped me was the HUUUUUUUGE amount of services they have, and the pricing on that. DynamoDB was half the price of Cloud Firestore, the free tier on most services was half that of GCP, and the offerings were insane. Need to host a website? Cloud Front. Launch an app? Lightsail. Satellite connection? Covered. Streaming Data? Kinesis. Queried Database? Elasticsearch. And that's just scratching the surface, I know.

Honestly, in a few years, I think we'll see Google forced to exit this space, Azure and AWS competing on price for big name corporate contracts, while others such as Digital Ocean being crumpled, simply because the AWS offering is so vast and widely supported online. And AWS' lock-in is pretty damn good.

The only solution to break out of such a duopoly would be for tech oriented companies to stop being lazy and start building out their own cloud infra.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#136

Post by santoshalper »

oblio wrote: post #114
ProAm wrote: post #64

Earlier quoted context omitted.

> I think the main advantage of cloud providers is to offset the risk of purchasing equipment that eventually is no longer needed, Every corporate use case I have seen is labor based. They dont want the overhead of salary and healthcare for the IT department. Even if long term they end up paying more, they always view it has pay for it now or pay for it later. And they always choose later because they dont know bette…

> And they always choose later because they dont know better. This is a dangerous assumption to make. Delaying payments and going with the crowd are both safe decisions. Safe decisions are smart decisions under normal circumstances.

Especially when IT is not your strategic differentiator.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#137

Post by dnautics »

hengheng wrote: post #100
ericbarrett wrote: post #61

Earlier quoted context omitted.

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of…

You still want ha, failover, and disaster recovery. Then you need to set up stuff like bgp, dns, security rules, etc, etc etc. Complexity mounts pretty quickly.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#138

Post by marta_morena_28 »

rualca wrote: post #122
waprin wrote: post #107

Earlier quoted context omitted.

It's not just managing the complexities of managing the bare metal, although that's certainly a huge component of it. There's some other huge arguments against running your own datacenters. One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need the…

> One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. A 75 million dollar price tag is also huge. Bothering about the operational impact of a VM or a request sent to a function-as-a-service might be a significant operational issue…

> 75 million dollar price tag is also huge.

Is it? Over the two years they saved, this is about the salary for 100 engineers. Can you replace and maintain all the cloud aspects that AWS provides you with (I mean the ones you actually use) with 100 engineers? Maybe, if they are good engineers (which is kinda implied by the 300k salary tag in the calculation). If it's worth it, remains to be seen. Definitely nothing for any medium sized company.

Dropbox is huge and has a relatively simple, highly optimized use-case, for which cloud perhaps doesn't offer too much. This is NOT the norm. For most companies, no matter the size, building their own cloud is a no go.

> those features are killed off and vanish from the face of the earth?

Don't use Google Cloud then ;).

> but if we consider the absurd premium charged by cloud providers for their services...

Do you have any data on backing this up? This "absurd premium" includes the salaries of engineers to develop it, maintain it, do DevOps, keep the hardware/data centers, do marketing, etc. etc. There is of course a margin, these companies aren't doing it as a social service... That margin is highly variable from service to service and also between cloud providers. Some may not have a margin at all, others may run at a loss. There is not easy "uh everything is overpriced". Most companies will have a VERY hard time providing the offering at the price of large cloud providers. And the simple "back of the envelope" calculations often miss all the work & cost that needs to be done, but you don't know about...

> Well, those "others" doing the work can be employed by your company and you still save money.

Yeah, if your company is really big, then yes. If your revenue is below 100 million, there isn't even room for any discussion on this: Don't run your own cloud, it's not gonna work. Most of the "cons" I see are about misunderstandings of the offerings and failure to navigate the pricing models and picking the cheapest offerings that do the job. If you fail to do even that, how on earth are you going to run your own cloud?

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#139

Post by dnautics »

dodobirdlord wrote: post #97
Ericson2314 wrote: post #89

Earlier quoted context omitted.

But it's insane that running a for-higher data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) In a sane economy, data centers for higher would be a fully-commoditized barely-profitable common carrier with little natural monopoly.

Reliability concerns make datacenters resilient to commoditization. A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. Commodity businesses are largely built on presumptions of linearity. Produce 90% as much corn/iron/wood/widgets as you expected and you’ll probably make something like 90% of the money you expected. Produce a 90% available datac…

> A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time.

That depends on your workload and the nature of the loss of availability. Is the completed work still there but just unreachable for an hour every few months? That might be okay for some folks.

Is your data center always available mon-fri but constantly has scheduled downtime on weekends? Might still be okay.

Post reply on HN