Building your own deep learning computer is 10x cheaper than AWS • Hacker News Board

Re: Building your own deep learning computer is 10x cheaper than AWS

#191

Post by cameldrv » Tue, Sep 25, 2018, 6:28 PM UTC

Great post. From someone who's done a few of these, I'll make a few observations:

1. Even if your DL model is running on GPUs, you'll run into things that are CPU bound. You'll even run into things that are not multithreaded and are CPU bound. It's valuable to get a CPU that has good single-core performance.

2. For DL applications, NVMe is overkill. Your models are not going to be able to saturate a SATA SSD, and with the money you save, you can get a bigger one, and/or a spinning drive to go with it. You'll quickly find yourself running out of space with a 1TB drive.

3. 64GB of RAM is overkill for a single GPU server. RAM has gone up a lot in price, and you can get by with 32 without issue, especially if you have less than 4 GPUs.

4. The case, power supply, and motherboard, and RAM are all a lot more expensive for a properly configured 4 GPU system. It makes no sense to buy all of this supporting hardware and then only buy one GPU. Buy a smaller PSU, less RAM, a smaller case, and buy two GPUs from the outset.

5. Get a fast internet connection. You'll be downloading big datasets, and it is frustrating to wait half a day while something downloads and you can get started.

6. Don't underestimate the time it will take to get all of this working. Between the physical assembly, getting Linux installed, and the numerous inevitable problems you'll run into, budget several days to a week to be training a model.

Re: Building your own deep learning computer is 10x cheaper than AWS

#192

Post by xienze » Tue, Sep 25, 2018, 6:33 PM UTC

vl wrote: post #190

drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

I have a hilarious story about this from Google: I wanted second 30" monitor, so I filed a ticket. They sent me long email listing reasons why I shouldn't get a second monitor, including (numbers are approximate, employee count from 2013 or so) "If every googler gets an extra monitor, in a year it would be equivalent to driving Toyota Camry for 18000 miles." I'm thinking "this can't possible be right", so I spend som…

> They sent me long email listing reasons why I shouldn't get a second monitor

I had no idea Google was so cheap. Gourmet breakfast, lunch, and dinner every day? No problem. A couple hundred bucks for a second monitor? Uh... it's not about the money, we're, uh, concerned about the environment.

Re: Building your own deep learning computer is 10x cheaper than AWS

#193

Post by smilliken » Tue, Sep 25, 2018, 6:33 PM UTC

mlrtime wrote: post #102

vidarh wrote: post #44

Earlier quoted context omitted.

I used to manage hardware in several datacentres, and I'd usually visit the data centres a couple of times a year . Other than that we used a couple of hours of "remote hands" services from the datacentre operator. Overall our hosting costs were about 30% of what the same capacity would have cost on AWS. Once a year I'd get a "why aren't we using AWS" e-mail from my boss, update our costing spreadsheets and tell him…

When you create the spreadsheet, do you price in running servers 24x7 or using elastic capacity?

(not the parent)

We've seen an even wider cost disparity on colo and dedicated servers vs AWS. More than 5x. It's easy for us to estimate because databases and web servers need to be on all the time, and those dominate costs.

Re: Building your own deep learning computer is 10x cheaper than AWS

#194

Post by ethbro » Tue, Sep 25, 2018, 6:33 PM UTC

supernovae wrote: post #135

whatshisface wrote: post #124

Earlier quoted context omitted.

The parent's point was that if the ops/deployment engineering team is unresponsive to the needs of developers, it may end up being better to run with no standards in cowboy-mode. If they ops team is extremely fast and highly skilled, they will be a boon full stop. If they are unskilled and politically obstructive, they will be a curse, full stop.

I got a more "you're doing it wrong" vibe...

I work at a Top25 US company by revenue.

In no way am I responsible for our migration going well. But I do feel if a company this size can do it, then others failing to do so says more about their architecture teams than the endeavor's futility.

Re: Building your own deep learning computer is 10x cheaper than AWS

#195

Post by riku_iki » Tue, Sep 25, 2018, 6:55 PM UTC

scottlegrand2 wrote: post #152

dagw wrote: post #89

Earlier quoted context omitted.

What are they putting into these cookies Cheap palm oil and other vegetable fats instead of butter, corn syrup instead of cane sugar, much lower quality chocolate than is generally available in grocery stores, various chemical stabilizers instead of eggs, artificial vanilla flavoring instead of real vanilla and so on. Plus they obviously get everything at a massive discount since they're buying everything by the ton…

You might want to ask why those cheap ingredients are cheap and you will find a bunch of farmers getting huge subsidies from the taxpayers to grow otherwise unprofitable crops. Kind of like how if you remove all the subsidies given to fossil fuel producers, some forms of renewable energy suddenly become much more competitive. So when you add a whole cost center for IT, the inefficiencies of the way they operate sudde…

I also think processed products usually require less effort to store and transport than some fresh ingredients.

Re: Building your own deep learning computer is 10x cheaper than AWS

#196

Post by Johnny555 » Tue, Sep 25, 2018, 7:08 PM UTC

drewg123 wrote: post #85

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

My company has the same policy - one monitor per employee. When I wanted a second one, I bought my own, set it up over the weekend, and when someone from IT finally noticed it a month or so later, I just said "Oh, I found it in the hardware e-waste pile over by the server room"

Not sure if I can take it with when I leave, but hey, at least now I have my second monitor and it cost less than $300 - I'll probably bequeath it to a fellow employee.

Re: Building your own deep learning computer is 10x cheaper than AWS

#197

Post by Johnny555 » Tue, Sep 25, 2018, 7:13 PM UTC

mynegation wrote: post #76

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

They seriously can’t. Typically if you want to buy some hardware in a big Corp you go to some preapptoved internal catalog to pick up from one or two models of desktop or a laptop from vendors like Dell, HP, or Lenovo - whatever was preapproved. Sometimes you can fine tune specs but not in a very wide range. Buying anything outside of that will require senior level approval and - heaven forbid - “vendor approval stud…

I worked at a startup that started doing that, the company was only around 250 people at the time, but the VP of IT was hired from a fortune 100 company, and he instituted the catalog approach of only 2 supported laptop configs (13" or 15"), and one desktop.

When I left the company, the engineering director was working out how to let his team go rogue - basically handling all of their own IT support, the only caveat being that they had to install IT's intrusion detection software, but aside from that, an engineer would be able to buy any machine within a certain budget.

Re: Building your own deep learning computer is 10x cheaper than AWS

#198

Post by paulmd » Tue, Sep 25, 2018, 7:22 PM UTC

spiritcat wrote: post #112

drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

Wow, no dual/triple monitor setup is just uncivilized.

At my employer, I happened to get one of the first machines they built with SSDs (in 2014) and one of the last machines they issued as desktop instead of laptop.

A year later they stopped issuing SSDs and I have to do the whole "managerial approval" thing to get any variation from the "standard" machine, so I am more than a year overdue for a replacement.

Re: Building your own deep learning computer is 10x cheaper than AWS

#199

Post by PascLeRasc » Tue, Sep 25, 2018, 7:22 PM UTC

nabla9 wrote: post #122

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

That's very passive aggressive way to deal with it. If that's your only option you are really a cubicle slave in corporate hell. In my opinion it's better to escalate upwards with proposals and not back down easily. You just have to frame it correctly and use right names and terms. * Usually big companies understand the concept of "a lab" that has infrastructure managed outside the corporate IT. Once you fight the ha…

I wanted to put an Ubuntu partition on my work PC for Python deep learning work, as I'm significantly faster and happier on it. When I mentioned it to the sysadmin, he said "I'm not allowing that. Linux is like Wikipedia, any idiot can contribute to it. Windows is made by professionals so it has to be better."

Re: Building your own deep learning computer is 10x cheaper than AWS

#200

Post by jimmy1 » Tue, Sep 25, 2018, 7:27 PM UTC

tapatio wrote: post #166

jimmy1 wrote: post #153

Earlier quoted context omitted.

Money^, benefits^^, ample PTO^^^, and a 8-5 workday allowing plenty of time for the important things in life like my family are exactly the reasons I do. ^: You know, like real money, not 90k plus 2.5% of something that will probably not exist next year ^^: You know, like real benefits such as low-cost, low-deduct. health insurance, matching 401k, etc. ^^^: As in, real, actual PTO, not this "unlimited" bullshit some…

Low deductible healthcare? Where do you live? Europe?

America, surprisingly, in a southern state no less, where heart disease and diabetes rates are higher. Our work has a grandfathered non-ACA plan (we got to keep our old insurance but means we miss out on "free" stuff like contraceptives and a breast pump, and other banal things, but in exchange, we get much better coverage, and a lower deductible)