Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are GPUs Worth It for ML?

exafunction.com

Post reply on HN
Print view View original

31–40 of 100 posts

Re: Are GPUs Worth It for ML?

#31

Post by skimo8 »

mushufasa wrote: post #27
beecafe wrote: post #25

Earlier quoted context omitted.

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

> You are years behind if you think you're training a model worth anything on consumer grade GPUs Ah yes, my code can't be useful to people unless it takes a long time to compile...

To be fair, I think ML workloads are quite a bit different than the days of compiling over lunch breaks.

What the above post was probably trying to get at is that the ML specific hardware is far more efficient these days than consumer GPUs.

Re: Are GPUs Worth It for ML?

#33

Post by ummonk »

What a clickbaity article. It’s an interesting discussion of GPU multiplexing for ML inference merged together with a sales pitch but the clickbait title made me hate the article bait and switch. This wasn’t even an example of Betteridge’s law but just completely misleading headline.

Re: Are GPUs Worth It for ML?

#34

Post by MichaelBurge »

PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

Think Google: Every time you search, some model somewhere gets invoked, and the aggregate inference cost would dwarf even very large training costs if you have billions of searches.

Marketing blogspam like this is always targeting big(not Google, but big) companies hoping to divert their big IT budgets to their coffers: "You have X million queries to your model every day. Imagine if we billed you per-request, but scaled the price so in aggregate it's slightly cheaper than your current spending."

People who are training-constrained are early-stage(i.e. correlate with not having money), and then they need to buy an entirely separate set of GPUs to support you(e.g. T4s are good for inference, but they need V100s for training). So they choose to ignore you entirely.

Re: Are GPUs Worth It for ML?

#36

Post by machinekob »

beecafe wrote: post #25
machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Ahh yes cause there is only one way to do Deep Learning and it is ofc stacking models large enough to not be useful outside pods of GPUs and this is for sure way to go if you want to make money (from VC ofc cause you wont have much users that are ever willing to pay so much that you'll ever make even, as was OpenAI and other big model providers, maybe you can get some money/sponsoring from state or uni).

Market for local small and efficient models running on device is pretty big maybe even biggest that exist right now [ios, android and macos are pretty easy to monetize with low cost models that are useful]. I can assure you of that and you can do it on even 4x RTX 3090 [ it wont be fast but you'll get there :) ]

Re: Are GPUs Worth It for ML?

#37

Post by scosman »

machinekob wrote: post #17
scosman wrote: post #9

We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.…

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

As mentioned in the comment, ML training workloads tend to be super chunky (at least in my experience). Some days we want to train 50 models, some weeks we are evaluating and don’t need any compute.

I’d rather be able to spin up 200 gpus in parallel when needed (yes, at a premium), but ramp to 0 when not. Data scientists waiting around are more expensive than GPUs. Replacing/maintaining servers is more work/money than you expect. And for us the training data was cloud native, so transfer/privacy/security is easier; nothing on prem, data scientists can design models without having access to raw data, etc.

Re: Are GPUs Worth It for ML?

#38

Post by machinekob »

scosman wrote: post #37
machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

As mentioned in the comment, ML training workloads tend to be super chunky (at least in my experience). Some days we want to train 50 models, some weeks we are evaluating and don’t need any compute. I’d rather be able to spin up 200 gpus in parallel when needed (yes, at a premium), but ramp to 0 when not. Data scientists waiting around are more expensive than GPUs. Replacing/maintaining servers is more work/money tha…

If you are cloud only company then for sure it is just easier but still it wont be cheaper just more convenient to use. If data science team is very big probably "best" solution without unlimited money is just to run local and go cloud [premium] if you dont have free resources for your teams (It was the case when i was working in pretty big EU bank but it wasn't "true" Deep learning yet [about 4-5 years ago]).

Re: Are GPUs Worth It for ML?

#39

Post by varunkmohan »

fancyfredbot wrote: post #35

There are some pretty elegant solutions out there for the problem of having the right ratio of CPU to GPU. One of the nicer ones is rCUDA. https://scholar.google.com/citations?view_op=view_citation&h...

rCUDA is super cool! One of the issues though is for a lot of the common model frameworks are not supported and a new release has not come out a while.

Re: Are GPUs Worth It for ML?

#40

Post by lajamerr »

mgraczyk wrote: post #22
PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

This depends a lot on what you're doing. If you are ranking 1M qps in a recommender system, then training cost will be tiny compared to inference.

I wonder if there's room for model caching. Sacrifice some personalization for near similar results so you aren't hitting the model so often.
Post reply on HN