Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building your own deep learning computer is 10x cheaper than AWS

medium.com

Post reply on HN
Print view View original

221–230 of 269 posts

Re: Building your own deep learning computer is 10x cheaper than AWS

#221

Post by Nimitz14 »

payne92 wrote: post #146

This analysis assumes 100% utilization: keeping the machine busy 24/7/265. It also ignores AWS price drops and the value of being able to switch to new AWS GPU instance types as they become available. TL;DR: Proposed machine costs $3k plus ~$100-200?/month for electricity, comparable AWS is currently $3/hr. My conclusion: If you're going to do more than 1-2,000hrs of GPU computing in the next few years, start thinkin…

If you just run something for 3 days every 2 weeks using your numbers that comes out to $432 per month. Just one month! Still not including storage costs etc. So after 6 months you're even. And you can still sell those GPUs in 1/2 years for a decent chunk of money when you want to buy the next generation.

The cloud only really makes sense for the on-demand burst capability IMO.

Re: Building your own deep learning computer is 10x cheaper than AWS

#222

Post by scosman »

Missing 1 important point: ML workflows are super chunky. Some days we want to train 10 models in parallel, each on a server with 8 or 16 GPUs. Most days we're building data sets or evaluating work, and need zero.

When it comes to inference, sometime you wanna ramp up thousands of boxes for a backfill, sometimes you need a few to keep up with streaming load.

Trying to do either of these on in-house hardware would require buying way too much hardware which would sit idle most of the time, or seriously hamper our workflow/productivity.

Re: Building your own deep learning computer is 10x cheaper than AWS

#223

Post by pkaye »

drewg123 wrote: post #85
VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

Reminds me of the time when LCD monitors were just taking over. The software team had budgeted and gotten approval for nice new development system along with a large LCD monitor. As we tried to place the order, the IT manager gets hold and makes a big fuss about it. Given that the CEO even approved it, it caught us off guard. Best we could tell she was butt hurt for us getting LCD monitors before their team did.

Re: Building your own deep learning computer is 10x cheaper than AWS

#224

Post by thrower123 »

spiritcat wrote: post #112
drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

Wow, no dual/triple monitor setup is just uncivilized.

When decent monitors cost like $100-$150, it's nuts. I've got a quad 27" setup. I'd like to have more, but things get a little dodgy when you run out of real video ouputs and have to start using displaylink usb adapters.

Re: Building your own deep learning computer is 10x cheaper than AWS

#225

Post by mmt »

LeonM wrote: post #3

Own hardware is always cheaper to buy than using a cloud service, but keeping it running 24/7 involves substantial costs. Sure, if you run a solo operation, you can just get up during the night to nurse your server, but at some point that no longer makes sense to do. Somewhere along the way we forgot about this and it's now perfectly normal to run a blog on a GKE 3 VM kubernetes cluster, costing 140 EUR/month.

> keeping it running 24/7 involves substantial costs

So does using a cloud service. It's not actually obvious, conceptually, but very little of the admin overhead has to do with the "own hardware" aspect of running it, especially if one excludes anything that has a direct analog at a cloud service.

There certainly exist services that abstract away more of this, but that's in exchange for higher cost and lower top performance, but that doesn't scale (in terms of cost).

> Sure, if you run a solo operation, you can just get up during the night to nurse your server, but at some point that no longer makes sense to do.

I'd actually argue the reverse. My experience is that the own-hardware portion took at most a quarter of my time, and that remained constant up to several hundred servers. It's much cheaper per unit of infrastructure the more units you have.

The tools and procedures that allow that kind of efficiency were the prerequisite for cloud services to exist.

Re: Building your own deep learning computer is 10x cheaper than AWS

#226

Post by da_chicken »

pythonaut_16 wrote: post #168
da_chicken wrote: post #134

Earlier quoted context omitted.

It's fine for exactly what the article describes: a student doing projects at home. For anything bigger than that, you'll quickly run into issues. Try talking to your ops team and telling them you want to set up a mid-range desktop PC with extra RAM and a high end graphics card as a DL workstation. I think you'll very quickly find some friction, especially once you want to go into production with it.

Google went to production with Pentium desktops on shelves. IIRC their original hardware plan was based around commodity hardware and getting the most performance per dollar since they built the crawler to be distributed pretty early on. Point being, consumer grade hardware has come a long way since then, and if you're doing something cutting edge like DL it's not outlandish to expect that rolling your own might be w…

That is a horrible comparison.

Google's start up between 1996 and 1998 was also over twenty years ago. There were fewer than 200 million Internet users on the Internet and less than 2.5 million web sites [0] in 1998. Google was also started by two college grad students, meaning gasp it was initially just a research project. I'd also point out that while, yes, the initial production servers were cheap and used commodity hardware, this [1] is what they looked like, which is hardly the type of setup that the article is suggesting.

[0]: http://www.internetlivestats.com/total-number-of-websites/

[1]: https://en.wikipedia.org/wiki/History_of_Google#/media/File:...

Re: Building your own deep learning computer is 10x cheaper than AWS

#227

Post by mmt »

mlrtime wrote: post #102
vidarh wrote: post #44

Earlier quoted context omitted.

I used to manage hardware in several datacentres, and I'd usually visit the data centres a couple of times a year . Other than that we used a couple of hours of "remote hands" services from the datacentre operator. Overall our hosting costs were about 30% of what the same capacity would have cost on AWS. Once a year I'd get a "why aren't we using AWS" e-mail from my boss, update our costing spreadsheets and tell him…

When you create the spreadsheet, do you price in running servers 24x7 or using elastic capacity?

And if considering elastic capacity, do you include the cost of the engineering effort required to take advantage of it?

A similar question applies to any other dynamic cost-reduction measure, such as spot instances.

I recall reading an announcement that GCP was starting only charging for actually-used vCPUs, rather than all that were provisioned, a form of automatic elastic cost-savings, although it was still more expensive than a DIY method. AFAIK, AWS doesn't do anything like that.

Re: Building your own deep learning computer is 10x cheaper than AWS

#228

Post by mmt »

kakwa_ wrote: post #136
maxehmookau wrote: post #108

While, in sheer dollar amount this post is probably correct, it doesn't really scale. At scale, you need more than just hardware. It's maintenance, racks, cooling, security, fire suppression etc. Oh, and the cost of replacing the GPUs when they die. At full price, yes, cloud GPUs on AWS aren't cheap, but at potentially a 90% saving in some regions/AZs, the price of spot instances by bidding on unused capacity for ML…

And the same can be said about can in fact pretty much be said by any AWS service. The equivalent of an i3 metal is probably around 30000 to 40000$ with Dell or HP, and probably half cheaper if self assembled (like a supermicro server). AWS i3.metal will cost 43000 annually, so even more than the acquisition cost of the server, server which will last probably around 5 year. But if you start taking into account all th…

> But if you start taking into account all the logistic, additional skills, people and processes needed to maintain a rack in a DC,

You've mostly described what one pays a datacenter provider, plus hiring someone who has experience working with one (and other own-hardware vendors, such as ISPs and VARs), which doesn't cost any more (and maybe less) than hiring someone with equivalent cloud vendor expertise.

> plus the additional equipment (network gears, KVMs, etc).

Although these are non-zero, they're a few hundred dollars (if that) per server, at scale, negligible compared to $20k.

> The cost win is far less evident

It still is, since the extra costs usually brought up are rarely quantified, and, when they are, turn out to be minor (nowhere near even doubling the cost of hardware plus electricity). AWS could multiply it by 10 (as in the very rough pricing example you provided).

> generally adds delays when product requirements changes.

This is cloud's biggest advantage, but it's not directly related to cost. This advantage can easily be mitigated by merely having spare hardware sitting idle, which is, essentially part of what one is paying for at a cloud provider.

Re: Building your own deep learning computer is 10x cheaper than AWS

#229

Post by [deleted] »

walshemj wrote: post #204
PascLeRasc wrote: post #199

Earlier quoted context omitted.

I wanted to put an Ubuntu partition on my work PC for Python deep learning work, as I'm significantly faster and happier on it. When I mentioned it to the sysadmin, he said "I'm not allowing that. Linux is like Wikipedia, any idiot can contribute to it. Windows is made by professionals so it has to be better."

And they call themselves a "sysadmin" !

[deleted]

Re: Building your own deep learning computer is 10x cheaper than AWS

#230

Post by poulsbohemian »

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

Could be very interesting when we have another economic downturn to see if attitudes on this change. It certainly seems more cost-effective to run ones own technical operations rather than offloading onto AWS / Google / Microsoft.
Post reply on HN