Building your own deep learning computer is 10x cheaper than AWS • Hacker News Board

Re: Building your own deep learning computer is 10x cheaper than AWS

#251

Post by candiodari » Wed, Sep 26, 2018, 10:58 AM UTC

solatic wrote: post #109

VHRanger wrote: post #70

Earlier quoted context omitted.

They seriously can't buy a graphics card and slap it in the PCIe slot?

You're neglecting management costs. IT teams don't buy hardware with corporate credit cards, they have to work through pre-existing requisition processes that properly budget for the hardware, make sure support contracts are in place, etc. You have to migrate whatever workload off the server where you installed the GPU (politically problematic since Murphy promises you that your users will be connecting to the server…

However, this cannot justify just any disparity in prices. Data science FTW !

Capex => Opex has a name. It's called "a loan". So let's model cloud usage as a loan. Let's assume you want, for an employee, a machine learning rig and you set depreciation on 2 years. Let's also assume that the article is correct about the ballpark figures, somewhere around $1400/month for renting, and $3000 for the machine.

And let's ignore the difference in power usage, extra space, bandwidth, ... (which is going to be 2 digit dollars at the very most anyway, since you need all that for the employee in the first place)

So how much interest does cloud charge for this Capex => Opex change ? Well 33600/30000.5 or 613.5% per year. Pretty much every bank on the planet will offer very low credit score companies 30% loans, even 10% is very realistic.

There are no words ... Just give the man his bloody machine. Hell, give him 5, 4 just in case 3 fail, and 1 for Crysis just to be a "nice" guy (you're not really being so nice: you're saving money) and you still come out ahead of the cloud.

We both know why this doesn't happen, the real reason: you don't trust your employees. Letting this employee have that machine would immediately cause a jealousy fight within the company, and cause a major problem. That's of course why the GP comment is right: leave this bloody nightmare of a company, today.

Re: Building your own deep learning computer is 10x cheaper than AWS

#252

Post by candiodari » Wed, Sep 26, 2018, 11:01 AM UTC

skywhopper wrote: post #149

So this is definitely an interesting article with some good ideas. But the main thrust isn't particularly interesting. It's always going to be true that building and operating your own is cheaper than using "the cloud"... so long as you are making use of it much of the time; and if you have the resources and facilities to build, operate, troubleshoot, and replace the hardware; and if you are sure about your long-term…

Except in this case, if you use it once a week, you're clearly coming out ahead.

That's getting pretty extreme.

Re: Building your own deep learning computer is 10x cheaper than AWS

#253

Post by benbristow » Wed, Sep 26, 2018, 11:51 AM UTC

w8rbt wrote: post #155

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

Yep. No one telling me no anymore and I can write Lambas to replace cron jobs, use RDS to replace DBs, use S3 and Glacier to replace Storage, etc. Fargate is awesome too. No gate keepers nor bureaucracy just code and git repos. That's why AWS is so awesome. And, I can show exactly how much everything costs. As well as work to reduce those costs. AWS added to a small, skilled dev team is a huge multiplier.

> use RDS to replace DBs

RDS is just managed databases though...

Re: Building your own deep learning computer is 10x cheaper than AWS

#254

Post by badpun » Wed, Sep 26, 2018, 2:33 PM UTC

solatic wrote: post #101

ethbro wrote: post #79

Earlier quoted context omitted.

Respectfully, those companies' cloud architects suck. If someone goes to the trouble to migrate onto cloud, and then replicates pre-devops workflows... wow.

A lot of those companies don't have cloud architects, just Amazon/Microsoft/Google sales reps talking into the ears of MBAs about turning capex into opex.

In my current company (a large bank), our team (~50 people) is trying to leave the on-premise infrastructure and move to the cloud because the on-perm stuff is managed in such a way that it's hard for us to acomplish anything. It will probably cost a lot more to use the cloud, but we're gladly willing to pay for it if we can shed the bureucracy this way.

I've read that in medieval times, kings have sometimes abandoned their castles for new ones after too much fecies were accumulated in them (as people were shitting wherever back then). I feel a strong parallel in this story to our situation.

Re: Building your own deep learning computer is 10x cheaper than AWS

#255

Post by tlear » Wed, Sep 26, 2018, 2:45 PM UTC

thisisjeffchen wrote: post #105

TeeWEE wrote: post #103

You only need such a machine for short time periods most of the time. So renting it is much more easy and cheaper.

Hi! Thank you for the comment, I'm the writer on the article. When we did our training we actually needed the computer for months at a time. It takes 1-2 months to tune a model and we were running exps almost 24/7. So one project put us in the break even point to build.

Was it possible to paralelize the process? Seems like training on few dozen cloud GPUs and finishing much faster would save a lot of money in terms of engineer time

Re: Building your own deep learning computer is 10x cheaper than AWS

#256

Post by treis » Wed, Sep 26, 2018, 4:23 PM UTC

unbearded wrote: post #165

dagw wrote: post #29

You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)

Unfortunately, the fight goes even further than that when you go against the cloud. Last week I was in an event with the CTOs of many of the hottest startups in America. It was shocking how much money is wasted on the cloud because inefficiencies and they simply don't care how much it costs. I guess since they are not wasting their own money, they can always come up with the same excuse: developers are more expensive…

>In 6 years, there were only 3 outages that weren't my fault.

How many where there that were your fault? And of those, how many would have been avoided by using Heroku?

>All at the cost of office rent ($1000) + FIOS ($359) + Cloudflare costs and S3 for images and backups.

What about the time spent creating and maintaining this infrastructure?

Re: Building your own deep learning computer is 10x cheaper than AWS

#257

Post by p1esk » Wed, Sep 26, 2018, 5:01 PM UTC

oneshot908 wrote: post #137

It's been this way since day 1. NVLINK remains the only real Tesla differentiator (although mini NVLINK is available on the new Turing consumer GPUs so WTFever). But because none of the DL frameworks support intra-layer model parallelism, all of the networks we see tend to run efficiently in data parallel because doing anything else makes them communication-limited, which they aren't because data scientists end up bu…

22,000-wide output sourced by a 4096-wide embedding

You will want to use hierarchical outputs in this case. Take a look at Hinton's 'Knowledge Distillation' paper.

Re: Building your own deep learning computer is 10x cheaper than AWS

#258

Post by p1esk » Wed, Sep 26, 2018, 5:07 PM UTC

cameldrv wrote: post #232

oneshot908 wrote: post #137

It's been this way since day 1. NVLINK remains the only real Tesla differentiator (although mini NVLINK is available on the new Turing consumer GPUs so WTFever). But because none of the DL frameworks support intra-layer model parallelism, all of the networks we see tend to run efficiently in data parallel because doing anything else makes them communication-limited, which they aren't because data scientists end up bu…

I think that the reason no one implements Krizhevsky's OWT (at least in normal training scripts, there's nothing stopping you from doing this in TensorFlow) is that the model parallelism in OWT is only useful where you have more weights than inputs/outputs to a layer. This was true for the FC layers in AlexNet, but hardly anyone uses large FC layers anymore.

Model parallelism is also useful in situation where your model (and/or your inputs) is so large that even with batch_size=1 it does not fit in GPU memory (especially if you're still using 1080Ti). However other techniques might help here (e.g. gradient checkpointing, or dropping parts of your graph to INT8).

Re: Building your own deep learning computer is 10x cheaper than AWS

#259

Post by Jack000 » Wed, Sep 26, 2018, 6:10 PM UTC

scosman wrote: post #222

Missing 1 important point: ML workflows are super chunky. Some days we want to train 10 models in parallel, each on a server with 8 or 16 GPUs. Most days we're building data sets or evaluating work, and need zero. When it comes to inference, sometime you wanna ramp up thousands of boxes for a backfill, sometimes you need a few to keep up with streaming load. Trying to do either of these on in-house hardware would req…

on the other hand, this comparison accounts for the full cost of the rig, while a realistic comparison should consider the marginal costs. Most of us need a pc anyways, and if you're a gamer the marginal cost is pretty close to zero.

Re: Building your own deep learning computer is 10x cheaper than AWS

#260

Post by unbearded » Wed, Sep 26, 2018, 7:35 PM UTC

spullara wrote: post #174

unbearded wrote: post #165

Earlier quoted context omitted.

Unfortunately, the fight goes even further than that when you go against the cloud. Last week I was in an event with the CTOs of many of the hottest startups in America. It was shocking how much money is wasted on the cloud because inefficiencies and they simply don't care how much it costs. I guess since they are not wasting their own money, they can always come up with the same excuse: developers are more expensive…

Only a Rails developer would think 6k requests per minute with 40ms latency is reasonable with all that hardware. If you rewrote it you probably only need 1 server but you will probably make an argument about how developer time is more valuable :)

The 6k/req with 40ms is just at the front door.

I'm talking about a real application here. With 100s of database and API calls on each web page load. I could make the whole thing in Golang or Scala and that would be at least one order of magnitude faster. But then I would have to throw away all the business knowledge that was added to the Rails app.

For instance, the slowest API call on the 40 ms is one that hits an ElasticSearch cluster with over 1 billion documents and is made on a Scala backend using Apache Thrift. There's a lot of caching but still, long tail and customization will kill caching at the top level.