Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are GPUs Worth It for ML?

exafunction.com

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Are GPUs Worth It for ML?

#81

Post by varunkmohan »

fomine3 wrote: post #76
pqn wrote: post #6

Earlier quoted context omitted.

Disclaimer: I work at Exafunction I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand. But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.

GCP still provides NVIDIA K80. I wonder is it still worth to hold.

I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.

Re: Are GPUs Worth It for ML?

#82

Post by varunkmohan »

gpm wrote: post #79
PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

Alpha zero used 5000 TPUs to generate games (inference only), and 16 to train the networks. The split definitely depends on what you're doing past developing/deploying. (Source: https://kstatic.googleusercontent.com/files/2f51b2a749a284c2... )

Completely agreed. For some of these large language models, it would take a long time before inference spend dominates training spend.

Re: Are GPUs Worth It for ML?

#83

Post by godelski »

cardine wrote: post #24
fartcannon wrote: post #19

Earlier quoted context omitted.

How do you train new models if your GPUs are being used for inference? I guess the training happens significantly less frequently? Forgive my ignorance.

We have different servers for each. But the split is usually 80%/20% for inference/training. As our product grows in usage the 80% number is steadily increasing. That isn't because we aren't training that often - we are almost always training many new models. It is just that inference is so computationally expensive!

Are you training new models from scratch or just fine tuning LLMs? I'm from the CV side and we tend to train stuff from scratch because we're still highly focused on finding new architectures and how to scale. The NLP people I know tend to use LLMs and existing checkpoints so their experiments tend to be a lot cheaper.

Not that anyone should think any aspect (training nor inference) is cheap.

Re: Are GPUs Worth It for ML?

#84

Post by potatoman22 »

beecafe wrote: post #25
machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Most models actually being used are linear regressions and decision trees

Re: Are GPUs Worth It for ML?

#85

Post by visarga »

rvz wrote: post #69

Not only the end result of these deep learning models can be tricked over a single pixel or get confused by malicious input and becomes useless, Deep Learning training, retraining, fine tuning on GPUs, TPUs, all running in a data center contribute significantly to burning up the planet and driving up costs which the models are just used for nothing but surveillance on our own data. If it doesn't work it has to be ret…

Why so negative? It's a waste only if the value provided is less than the cost. You can't decide that with only the cost.

As humans we have our own adversarial examples, we get tired, we get sloppy, we might be even more biased than a calibrated model and always much more expensive.

Re: Are GPUs Worth It for ML?

#87

Post by thebruce87m »

beecafe wrote: post #25
machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

My model takes 6 hours to train on a 3090. People have different use cases.

Re: Are GPUs Worth It for ML?

#88

Post by fragmede »

macksd wrote: post #72
beecafe wrote: post #25

Earlier quoted context omitted.

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Isn't a key selling point of the latest, hottest model that's on the front page of Hacker News multiple times right now, the fact that it fits on consumer-grade GPUs? Surely some of the interesting ideas it's spawning right now are people doing transfer learning on GPUs that don't end in "100", don't you think?

for what it's worth, stable diffusion was trained on 32 x 8 x A100 GPUs

Re: Are GPUs Worth It for ML?

#89

Post by rvz »

visarga wrote: post #85
rvz wrote: post #69

Not only the end result of these deep learning models can be tricked over a single pixel or get confused by malicious input and becomes useless, Deep Learning training, retraining, fine tuning on GPUs, TPUs, all running in a data center contribute significantly to burning up the planet and driving up costs which the models are just used for nothing but surveillance on our own data. If it doesn't work it has to be ret…

Why so negative? It's a waste only if the value provided is less than the cost. You can't decide that with only the cost. As humans we have our own adversarial examples, we get tired, we get sloppy, we might be even more biased than a calibrated model and always much more expensive.

> Why so negative? It's a waste only if the value provided is less than the cost.

It is entirely true and it just takes an invalid input to trick them and it messes up easily and even worse when there are always biases involved. Thus the value is nullified.

And once that model breaks and doesn't work, what is the solution? More retraining on new data? Even with that like I said there are ZERO efficient alternatives, which the cost outweighs the benefits.

Therefore, it is not even worth it.

Re: Are GPUs Worth It for ML?

#90

Post by ramoz »

scosman wrote: post #9

We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.…

We did a similar analysis for GCP. Preemptibles/spot were the way to go with inference. CPU performance was also faster for our scaled inference workloads.

Times change though, we’re about to conduct the same analysis over again, with latest models better architected for accelerators.

Post reply on HN