Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dropbox saved $75M over two years by building its own infrastructure (2018)

geekwire.com

Post reply on HN
Print view View original

211–218 of 218 posts

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#211

Post by spullara »

einpoklum wrote: post #207
spullara wrote: post #189

Earlier quoted context omitted.

I was talking about at scale, not a rack. If you can get by with a rack, you will pay more for the people to support it than the incremental cost of the cloud.

> If you can get by with a rack, you will pay more for the people to support it than the incremental cost of the cloud. Probably a whole lot less. At larger scale - I would guess it's the same thing. If an organization needs more than a rack during peak use, it can probably benefit from setting up its own infrastructure. Only in the uncommon case of short extreme peak use and almost no use most of the time does such…

That is very common with the infrastructure startups that I work with, like Snowflake and others.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#212

Post by rstuart4133 »

ericbarrett wrote: post #61
calderarrow wrote: post #3

I think this quote really hits the nail on the head and confirms what a lot of people may have intuitively known about the value of cloud providers: But once certain startups turn into big companies with hundreds of millions of users, with computing needs that they’ve come to intimately understand, it can be far more efficient to set up computing infrastructure designed exactly with those needs in mind. I think the m…

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to…

> The problem will be finding skilled labor.

I think that overstates the problem. It does not require a whole pile of skill to purchase a few rack mount servers from Dell or Supermicro with onsite 24 hour warranty, and plonk them in a co-lo. In the rare event the hardware does break ring Dell and ask them to fix it for you. When the onsite warranty end's in 7 years it's time to replace the servers.

The expertise required it literally minimal - not much beyond the ability to use a screw driver to install it into the rack and know how to connect the Ethernet cables. Then you have to plug in a USB and install whatever OS you want, of course, but you don't have to be onsite to do that. They all come with iDRAC's or the equivalent.

They will cost about $1000/yr with maybe 10TB raid disk, co-lo costs of around another $1000/yr for unlimited bandwidth. To rent the same dedicated metal is about $500/mo from OHV where I live, so over twice the cost.

Obviously, this is all impossible if you aren't big enough to have dedicated IT staff. And obviously, if you are likely to go through rapid change (well, something more extreme than adding a new server every now and then), it isn't the best plan. But for a stable mature business that employs several hundred people, all you are really doing is cutting out the middle man.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#213

Post by acoard »

Daishiman wrote: post #163
acoard wrote: post #151

Earlier quoted context omitted.

>Definitely nothing for any medium sized company. I work at a medium sized company. Depends on who you count, but let's say around ~30 devs. Recently we basically did just this, and it's been a great success. We haven't fully migrated and still use AWS for prod, but have seen substantial savings already. We spent $2k on servers, Dell r720s. We bought a UPS and mount, and racked them in our office. I installed OpenShi…

This post proves parent's point though. You're not doing anything even remotely close to the features offered by cloud providers or even managed hosting providers. Disaster recovery? Geographically separate redundant servers with failovers? Automated (and proven to work) backups? One-stop access control for infra maintenance? Audit controls for your database and storage objects? Tape backups? Even today to support al…

>Disaster recovery? Geographically separate redundant servers with failovers? Automated (and proven to work) backups? One-stop access control for infra maintenance? Audit controls for your database and storage objects? Tape backups?

These are our dev+test setups, and we're looking far more carefully at prod for the reasons you touch on. Those aren't necessary for every project too, eg hosting computer vision demos.

For our government projects, the government hosts it on their own OpenShift cluster that they maintain (including their own data centre), due to requirements for all data to be hosted within our boarders. The OpenShift cluster I setup is no-where near as well maintained as the governments, they have multiple FTE and it runs most of the open source gov't code. They have tape backups, rolling on-call staff, public developer chat for support, the whole deal.

What I setup is far more simple. We have daily/weekly/monthly rolling backups of postgres pods. We store some backups of those on digital ocean, but that's just a cheapo litttle linux server.

But now a team of 30 developers can easily spin up their own projects using a web-based GUI from basically just providing a Dockerfile or a link to a git repo. One of the oft-touted organizational benefits of "cloud" is that you don't have to wait a week for Ops to provision a VM. We get all that.

>I understand your use case, but your is very, very far from the sheer and absolute complexity and features that enterprise data centers have.

My point is that many things people host in AWS do not need enterprise quality. If you're a startup, then almost by definition you do not need enterprise quality (though, as always, it depends). We made a tonne of savings. I'm sure many others would by self-hosting and learning a moderate amount of Linux / Kubernetes.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#214

Post by einpoklum »

spullara wrote: post #211
einpoklum wrote: post #207

Earlier quoted context omitted.

> If you can get by with a rack, you will pay more for the people to support it than the incremental cost of the cloud. Probably a whole lot less. At larger scale - I would guess it's the same thing. If an organization needs more than a rack during peak use, it can probably benefit from setting up its own infrastructure. Only in the uncommon case of short extreme peak use and almost no use most of the time does such…

That is very common with the infrastructure startups that I work with, like Snowflake and others.

Don't the various clients even out the usage?

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#215

Post by Shorel »

ericbarrett wrote: post #172
Shorel wrote: post #113

Earlier quoted context omitted.

You say "stubborn" but when the savings can be in the millions of dollars, it becomes logical to own a datacenter. I can't believe datacenter people suddenly all disappeared or are retired. If the pay scale is down, they will surely appreciate working for a big company with datacenter needs. Obviously, this only applies to huge companies like Dropbox. Everyone else is better served by AWS.

This is explicitly what I’m calling out at the end. They didn’t all disappear or retire; it’s an ongoing process. Think COBOL programmers.

Ha!

I was also thinking about COBOL programmers when writing my comment.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#216

Post by p_l »

igor_a wrote: post #26

$75M over two years is compensation of 75 engineers in the SV (not even taking into account opportunity cost - this 75 folks could innovate and bring more revenue) So either it's not $75m or it was a bad idea.

Or maybe the workload of using their own infrastructure was lower, and scales well, that the savings could go towards having 75 extra people who could innovate instead of not having them.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#217

Post by p_l »

filmgirlcw wrote: post #155
dralley wrote: post #66

Earlier quoted context omitted.

Doesn't Netflix have cache boxes hooked directly into the ISP networks? I'm sure Netflix uses a lot of AWS, but as a percentage of their total traffic (which is absolutely enormous), it's probably not that much.

Yes. And although much of what Netflix does is in-house, it’s done the math and realized that for some of its workloads, AWS is just better. Especially when looking at all of the areas where Netflix is available and needs datacenters and in some of the specific compute options Netflix needs. It’s not serving traffic through AWS. It’s using AWS for storage and compute.

It also certainly doesn't pay by the public price book.

Re: Dropbox saved $75M over two years by building its own infrastructure (2018)

#218

Post by Schwolop »

randomdude402 wrote: post #190
RemingtonLak wrote: post #181

Well unfortunately I'm seeing this in my microscopic way. I moved off all of my domains and hosting from GoDaddy (shrugging in disgust) to Google domain+GCE. yikes, with virtually no traffic outside of me ssh'ing, my 2 vm instances is costing me $45/mon just basically sitting idle. I gotta be doing something wrong!

You can spin up a VPS on digital ocean for $5/month

[off-topic] Hey randomdude - a few months back you replied to one of my comments with some well wishes that I could get back into motorbike riding soon. Just wanted to pass on that Melbourne is now free from lockdown again, and so I am able, and have been. It's great! I hope you're doing well too!
Post reply on HN