Building your own deep learning computer is 10x cheaper than AWS • Hacker News Board

Re: Building your own deep learning computer is 10x cheaper than AWS

#101

Post by solatic » Tue, Sep 25, 2018, 1:28 PM UTC

ethbro wrote: post #79

mlthoughts2018 wrote: post #45

Earlier quoted context omitted.

In most companies, AWS just becomes a new front-end to the same old IT bureaucracy, and dev teams are still disallowed from creating their own instances or EMR clusters or setting up new products like their own Redshift instance or ECR deployment solution.

Respectfully, those companies' cloud architects suck. If someone goes to the trouble to migrate onto cloud, and then replicates pre-devops workflows... wow.

A lot of those companies don't have cloud architects, just Amazon/Microsoft/Google sales reps talking into the ears of MBAs about turning capex into opex.

Re: Building your own deep learning computer is 10x cheaper than AWS

#102

Post by mlrtime » Tue, Sep 25, 2018, 1:29 PM UTC

vidarh wrote: post #44

LeonM wrote: post #3

Own hardware is always cheaper to buy than using a cloud service, but keeping it running 24/7 involves substantial costs. Sure, if you run a solo operation, you can just get up during the night to nurse your server, but at some point that no longer makes sense to do. Somewhere along the way we forgot about this and it's now perfectly normal to run a blog on a GKE 3 VM kubernetes cluster, costing 140 EUR/month.

I used to manage hardware in several datacentres, and I'd usually visit the data centres a couple of times a year . Other than that we used a couple of hours of "remote hands" services from the datacentre operator. Overall our hosting costs were about 30% of what the same capacity would have cost on AWS. Once a year I'd get a "why aren't we using AWS" e-mail from my boss, update our costing spreadsheets and tell him…

When you create the spreadsheet, do you price in running servers 24x7 or using elastic capacity?

Re: Building your own deep learning computer is 10x cheaper than AWS

#103

Post by TeeWEE » Tue, Sep 25, 2018, 1:31 PM UTC

You only need such a machine for short time periods most of the time. So renting it is much more easy and cheaper.

Re: Building your own deep learning computer is 10x cheaper than AWS

#104

Post by thisisjeffchen » Tue, Sep 25, 2018, 1:31 PM UTC

acd10j wrote: post #92

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

Most of time standard power supplies and cabinet of these Desktop does not support 1080Ti power consumption.

Hey there, I'm the writer for the article. I have another article planned that will talk about part picking and actual building. For now though, here is the parts list: https://pcpartpicker.com/b/B6LJ7P w/ 1600W psu.

Re: Building your own deep learning computer is 10x cheaper than AWS

#105

Post by thisisjeffchen » Tue, Sep 25, 2018, 1:34 PM UTC

TeeWEE wrote: post #103

You only need such a machine for short time periods most of the time. So renting it is much more easy and cheaper.

Hi! Thank you for the comment, I'm the writer on the article. When we did our training we actually needed the computer for months at a time. It takes 1-2 months to tune a model and we were running exps almost 24/7. So one project put us in the break even point to build.

Re: Building your own deep learning computer is 10x cheaper than AWS

#106

Post by bogomipz » Tue, Sep 25, 2018, 1:35 PM UTC

>"Nvidia contractually prohibits the use of GeForce and Titan cards in datacenters. So Amazon and other providers have to use the $8,500 datacenter version of the GPUs, and they have to charge a lot for renting it."

I wonder if someone might provide some clarification on this. Is this to say only if a reseller buys directly from Nvidia they are compelled by some agreement they signed with Nvidia? How else would this legal for Nvidia to dictate how and where someone is allowed to use their product? Thanks.

Re: Building your own deep learning computer is 10x cheaper than AWS

#107

Post by xfitm3 » Tue, Sep 25, 2018, 1:36 PM UTC

grogenaut wrote: post #100

drewg123 wrote: post #85

Earlier quoted context omitted.

When I worked at Google, I really missed the dual monitor setup I had had at my previous job. I asked my manager how to get a 2nd monitor. Apparently, since I had the larger monitor, I was not allowed to get a 2nd one without all kinds of hassle. I asked if I was allowed to just buy one from Amazon and plug it in, and I was told no. I finally just grabbed an older one that had been sitting in the hallway of the cube-…

When I started at Amazon they gave us dual 22" monitors. I bought dual 27" dells and a gfx card to back them and plugged it in. I also explained how much I was swapping with 8gb and a virt (45 minutes of lost productivity a day), and ram was $86. Manager happily expensed ram for me and the rest of the team . 2 years later that was the standard setup. Now I have 7 monitors and 3 PC's from multiple projects and oses an…

TIL Bias for action.

Re: Building your own deep learning computer is 10x cheaper than AWS

#108

Post by maxehmookau » Tue, Sep 25, 2018, 1:36 PM UTC

While, in sheer dollar amount this post is probably correct, it doesn't really scale.

At scale, you need more than just hardware. It's maintenance, racks, cooling, security, fire suppression etc. Oh, and the cost of replacing the GPUs when they die.

At full price, yes, cloud GPUs on AWS aren't cheap, but at potentially a 90% saving in some regions/AZs, the price of spot instances by bidding on unused capacity for ML tasks that can be split over multiple machines make using cloud servers a much more attractive prospect.

I think this post is conflating one physical machine to a fleet of virtualised ones, and that's not really a fair comparison.

Also, the post refers to cloud storage at $0.10/GB/month which is incorrect. AWS HDD storage is $0.025/GB/month and S3 storage is $0.023 which is arguably more suited to storing large data sets.

Re: Building your own deep learning computer is 10x cheaper than AWS

#109

Post by solatic » Tue, Sep 25, 2018, 1:42 PM UTC

VHRanger wrote: post #70

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

They seriously can't buy a graphics card and slap it in the PCIe slot?

You're neglecting management costs. IT teams don't buy hardware with corporate credit cards, they have to work through pre-existing requisition processes that properly budget for the hardware, make sure support contracts are in place, etc. You have to migrate whatever workload off the server where you installed the GPU (politically problematic since Murphy promises you that your users will be connecting to the server by its IP address, or the workload is "mission critical" and can't be stopped, or whatever) and reserve the server for GPU work. If you're running something like vCenter to virtualize your datacenter resources, you need to make sure that VMware picks up on the GPU in the new machine, doesn't schedule new VMs without GPU requirements on the server with the new GPU, etc.

Comments like these (why can't I just do X?), it's like the difference between being single and being in a serious relationship. When you're single, you can do whatever you want. When you're in a serious relationship, you can't just make whatever decisions you want without talking to your partner. Well, big corporate enterprise is like that on steroids, because instead of having one partner, you have several or dozens, and everyone needs to buy in.

Re: Building your own deep learning computer is 10x cheaper than AWS

#110

Post by thisisjeffchen » Tue, Sep 25, 2018, 1:42 PM UTC

soneil wrote: post #30

badestrand wrote: post #20

Earlier quoted context omitted.

According to the article you can break even after only two weeks which is far from "10k cookies every day for multiple years".

I think the analogy holds. The numbers in the article assume constant usage (there is a nod to the cost of static storage, but all the math is done on $3/hr gpu usage, not $0.1/mo/GB storage). So if your usage is 4 hours per week, 50 weeks a year, you hit $600/year of gpu. If your usage is 24 hours a day, you hit $27k. So it's not a clear-cut "buying metal is cheaper" - you have to decide how your usage amortizes. As…

Depends on your use case. The more accurate models these days require lots of data with huge models. Machine Translation, Image Generation, Segmentation, etc. Each experiment takes a couple days/weeks to train, so we were encountering long stretches of constant usage.