Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox saved $75M over two years by building its own infrastructure (2018)

geekwire.com

Post reply on HN
Print view View original

151–160 of 218 posts

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#151

Post by acoard »

marta_morena_28 wrote: post #138
rualca wrote: post #122

Earlier quoted context omitted.

> One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. A 75 million dollar price tag is also huge. Bothering about the operational impact of a VM or a request sent to a function-as-a-service might be a significant operational issue…

> 75 million dollar price tag is also huge. Is it? Over the two years they saved, this is about the salary for 100 engineers. Can you replace and maintain all the cloud aspects that AWS provides you with (I mean the ones you actually use) with 100 engineers? Maybe, if they are good engineers (which is kinda implied by the 300k salary tag in the calculation). If it's worth it, remains to be seen. Definitely nothing fo…

>Definitely nothing for any medium sized company.

I work at a medium sized company. Depends on who you count, but let's say around ~30 devs.

Recently we basically did just this, and it's been a great success. We haven't fully migrated and still use AWS for prod, but have seen substantial savings already.

We spent $2k on servers, Dell r720s. We bought a UPS and mount, and racked them in our office. I installed OpenShift 4 on it, which is Red Hat's Kubernetes offering with a nice web GUI, and setup a few terabytes of NFS to automatically provision storage.

To be fair, installing OpenShift for the first time took a while, around 3 weeks. Since then it's been smooth. We still use AWS, but our usage has gone down dramatically. We are still only migrating dev and test environments, leaving prod in AWS (we don't want to be responsible for uptime SLAs, and clients pay prod hosting costs). Some of these projects are CPU heavy, machine learning and computer vision projects too. They're not just simple web-apps. I'm not privy to our entire AWS budget, but I know that one project which we migrated saved over $500/mo.

After installation, maintenance has taken barely any time. Around 10-20% of my time is dedicated to OpenShift cluster maintenance. The rest I do normal project work. I often go weeks without having to touch anything, and the most common task I do is onboard new users. We've had 2 outages in over 6mo, one was an expiring cert and one was an airflow issue on the rack. I've learnt a lot and am certainly not an expert. These were the firs rack servers I'd ever worked with personally, although I had been researching used models for home use for a while (shoutout to /r/Homelab).

In fact, I had such success doing this that I personally bought a Dell r720 and have used it to selfhost a bunch of stuff at home. A co-worker of mine hosts his self hosted lab on AWS. Things like Plex, private photo storage, a few other toys, etc. He says he pays $300/mo, which seems insane to me, but I guess people streaming 4K plex adds up. The used r720 server I bought was $1,500CAD and has way more horsepower than he's paying for. (There are also electricity costs I haven't factored in here, as I'm trying to control for other changes in my power bill. Might be $100/mo at most.)

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#152

Post by jedberg »

dec0dedab0de wrote: post #132
benhoyt wrote: post #82

Earlier quoted context omitted.

Not a bad analogy, but the taxi also comes with a driver, which makes it expensive but allows you to get work done in the back while you're en route. It'd be like a cloud provider that came with a devops person dedicated to you.

Sure, but if you are using the taxi often enough it may be cheaper to higher a full time driver.

Assuming you are qualified to hire drivers. And know where to find a good one. This is what people forget with the cloud. That you're paying for their expertise.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#153

Post by ryanlol »

data_spy wrote: post #143
ebiester wrote: post #6

Just as a reminder: You are not Dropbox. You personally need to run the numbers with both scenarios. Depending on your use case, it may be cheaper to be in AWS, and it may be cheaper to be co-located. If you are as big as Dropbox, it may be cheaper to build your own data center. It reminds me of the arguments for and against K8s. Most of the discussion is based on use case, and not considering the solution space for…

Agreed. I work at a company that does 150 Billion transactions a day and it is cheaper being in the cloud than on prem for us. Maybe the variable cost is more but the fixed cost is a lot less for the business.

Cloud vs on prem is a ridiculous comparison. The sweet spot is almost always in the middle.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#154

Post by mike_d »

ebiester wrote: post #6

Just as a reminder: You are not Dropbox. You personally need to run the numbers with both scenarios. Depending on your use case, it may be cheaper to be in AWS, and it may be cheaper to be co-located. If you are as big as Dropbox, it may be cheaper to build your own data center. It reminds me of the arguments for and against K8s. Most of the discussion is based on use case, and not considering the solution space for…

> ...and it may be cheaper to be co-located. If you are as big as Dropbox, it may be cheaper to build your own data center.

Nothing I have seen indicates dropbox "built its own datacenter." It looks like they are in Equinix, CoreSite, and DRT colocation.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#155

Post by filmgirlcw »

dralley wrote: post #66
clubdorothe wrote: post #54

Wondering why a company like Netflix did the opposite?

Doesn't Netflix have cache boxes hooked directly into the ISP networks? I'm sure Netflix uses a lot of AWS, but as a percentage of their total traffic (which is absolutely enormous), it's probably not that much.

Yes. And although much of what Netflix does is in-house, it’s done the math and realized that for some of its workloads, AWS is just better. Especially when looking at all of the areas where Netflix is available and needs datacenters and in some of the specific compute options Netflix needs. It’s not serving traffic through AWS. It’s using AWS for storage and compute.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#156

Post by jgalt212 »

Riddle me this: computer and storage prices keep going down, but cloud spend keeps going up. How does that work out?

And more importantly, I won't go all in on the could until the big cloud vendors start running their workloads on machines they don't physically control.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#158

Post by cobookman »

llarsson wrote: post #57

Cloud is taking a taxi everywhere, bare metal is owning your own car. In one case, you pay a premium so that somebody else worries about all details and maintenance, and in the other, you take on that burden but don't pay the premium. This has always been true for everything where a service is provided to you. Great if your needs are small or not well known because there is no initial investment, but bad in the long…

A company could hire their own janitor, buy their own janitorial supplies and save money.

Or they can go through a 3rd party consultant to take care of it so they can focus their cycles on more important issues.

Not everyone is an infrastructure provider. Time is a limited commodity, and there are clear benefits of spending as much of your time on the core business.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#159

Post by mgiampapa »

ericbarrett wrote: post #61
calderarrow wrote: post #3

I think this quote really hits the nail on the head and confirms what a lot of people may have intuitively known about the value of cloud providers: But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. I think the m…

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

Some of it is becoming more democratized too. Facebook as an example open sources it's datacenter design, it's servers, switches etc. I know they are the big bad for privacy these days, but the OCP project has contributed state of the art datacenter tech that legacy providers should be jumping at.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#160

Post by einpoklum »

spullara wrote: post #103
hengheng wrote: post #100

Earlier quoted context omitted.

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of…

There are now businesses that explicitly depend on the elasticity of the cloud and can never really be moved on premise without massive up-front investment in hardware that may only be used a few times a year for their biggest customers. Trying to hybridize these workloads hasn't been very successful as of yet. It is possible that K8S could relive this problem but I haven't seen it in practice, at scale.

Elasticity? Fine. So their, say, single rack will sometime have limited load and be under-utilized.

About the up-front investment - most hi-tech companies are a massive initial up-front (or nearly-up-front) investment.

Post reply on HN