Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are GPUs Worth It for ML?

exafunction.com

Post reply on HN
Print view View original

41–50 of 100 posts

Re: Are GPUs Worth It for ML?

#41

Post by cjbgkagh »

beecafe wrote: post #25
machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Years behind what? Table stakes for what? There is much more to ML than the latest transformer and diffusion models. While those get the attention the amount of research not in that space dominates.

Re: Are GPUs Worth It for ML?

#42

Post by gowld »

PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

More to the point, you don't so training and inference in the same program, so don't have to be on the same hardware in the same machine. It's two separate problems with separate hardware solutions.

Re: Are GPUs Worth It for ML?

#43

Post by Frost1x »

mistrial9 wrote: post #13
Kukumber wrote: post #3

An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment

the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here

In what context? HPC and certain code bases have been effectively leveraging heterogenous CPU GPU workloads for a variety of applications for quite awhile. I know of some doing so in at least 2009 and know plenty of prior art was already there by that point, it's just a specific time I happen to remember.

Re: Are GPUs Worth It for ML?

#45

Post by PeterisP »

varunkmohan wrote: post #12
PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

Agreed that there are workloads where inference is not expensive, but it's really workload dependent. For applications that run inference over large amounts of data in the computer vision space, inference ends up being a dominant portion of the spend.

The way I see it, generally every new data point (on which the production model inference gets run once) becomes part of the data set which then gets used in training every next model, processing the same data point many more times in training, thus training unavoidably taking more effort than inference.

Perhaps I'm a bit biased towards all kinds of self-supervised or human-in-the-loop or semi-supervised models, but the notion of discarding large amounts of good domain-specific data that get processed only for inference and not used for training afterward feels a bit foreign to me, because you usually can extract an advantage from it. But perhaps that's the difference between data-starved domains and overwhelming-data domains?

Re: Are GPUs Worth It for ML?

#46

Post by dheera »

cardine wrote: post #18
PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

I have not found this to be true at all in my field (natural language generation). We have a 7 figure GPU setup that is running 24/7 at 100% utilization just to handle inference.

Also true of self-driving. You train a perception model for a week and then log millions of vehicle-hours on inference.

Re: Are GPUs Worth It for ML?

#47

Post by fortysixdegrees »

varunkmohan wrote: post #15
scosman wrote: post #9

We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.…

Disclaimer: I'm the Cofounder / CEO at Exafunction That's a great point. We'll be addressing this in an upcoming post as well. We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for…

Disclaimer != Disclosure

Probably one of HNs most common mistakes in comments

Re: Are GPUs Worth It for ML?

#48

Post by mgraczyk »

lajamerr wrote: post #40
mgraczyk wrote: post #22

Earlier quoted context omitted.

This depends a lot on what you're doing. If you are ranking 1M qps in a recommender system, then training cost will be tiny compared to inference.

I wonder if there's room for model caching. Sacrifice some personalization for near similar results so you aren't hitting the model so often.

Yeah we did lots of things like this at Instagram. Can be very brittle and dangerous though to share any caching amongst multiple users. If you work at Facebook you can search for some SEVs related to this lol

Re: Are GPUs Worth It for ML?

#49

Post by pdpi »

PeterisP wrote: post #45
varunkmohan wrote: post #12

Earlier quoted context omitted.

Agreed that there are workloads where inference is not expensive, but it's really workload dependent. For applications that run inference over large amounts of data in the computer vision space, inference ends up being a dominant portion of the spend.

The way I see it, generally every new data point (on which the production model inference gets run once) becomes part of the data set which then gets used in training every next model, processing the same data point many more times in training, thus training unavoidably taking more effort than inference. Perhaps I'm a bit biased towards all kinds of self-supervised or human-in-the-loop or semi-supervised models, but…

There's one piece of the puzzle you're missing: field-deployed devices.

If I play chess on my computer, the games I play locally won't hit the Stockfish models. When I use the feature on my phone that allows me to copy text from a picture, it won't phone home with all the frames.

Re: Are GPUs Worth It for ML?

#50

Post by rfrey »

Not related to the article, but how would one begin to become smart on optimizing GPU workloads? I've been charged with deploying an application that is a mixture of heuristic search and inference, that has been exclusively single-user to this point.

I'm sure every little thing I've discovered (e.g. measuring cpu/gpu workloads, trying to multiplex access to the gpu, etc) was probably covered in somebody's grad school notes 12 years ago, but I haven't found a source of info on the topic.

Post reply on HN