Are GPUs Worth It for ML? • Hacker News Board

fortysixdegrees wrote: post #47

varunkmohan wrote: post #15

Earlier quoted context omitted.

Disclaimer: I'm the Cofounder / CEO at Exafunction That's a great point. We'll be addressing this in an upcoming post as well. We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for…

Disclaimer != Disclosure Probably one of HNs most common mistakes in comments

perhaps, but I think disclaimer in this context it's just an abbreviation since the disclosure carries with it the implicit disclaimer of "so the things I'm saying are subconsciously influenced by the fact that they potentially could make me money"

Re: Are GPUs Worth It for ML?

#53

Post by pqn » Mon, Aug 29, 2022, 9:36 PM UTC

rfrey wrote: post #50

Not related to the article, but how would one begin to become smart on optimizing GPU workloads? I've been charged with deploying an application that is a mixture of heuristic search and inference, that has been exclusively single-user to this point. I'm sure every little thing I've discovered (e.g. measuring cpu/gpu workloads, trying to multiplex access to the gpu, etc) was probably covered in somebody's grad school…

Let's just take the topic of measuring GPU usage. This alone is quite tricky -- tools like nvidia-smi will show full GPU utilization even if not all SMs are running. And also the workload may change behavior over time, if for instance inputs to transformers got longer over time. And then it gets even more complicated to measure when considering optimizations like dynamic batching. I think if you peek into some ML Ops communities you can get a flavor of these nuances, but not sure if there are good exhaustive guides around right now.

Re: Are GPUs Worth It for ML?

#54

Post by Eridrus » Mon, Aug 29, 2022, 9:52 PM UTC

ummonk wrote: post #33

What a clickbaity article. It’s an interesting discussion of GPU multiplexing for ML inference merged together with a sales pitch but the clickbait title made me hate the article bait and switch. This wasn’t even an example of Betteridge’s law but just completely misleading headline.

Is everyone with relevant inference costs not doing this already?

I am so confused how there seems to be a startup around having a work queue that does batching...

Re: Are GPUs Worth It for ML?

#55

Post by varunkmohan » Mon, Aug 29, 2022, 10:00 PM UTC

PeterisP wrote: post #45

varunkmohan wrote: post #12

Earlier quoted context omitted.

Agreed that there are workloads where inference is not expensive, but it's really workload dependent. For applications that run inference over large amounts of data in the computer vision space, inference ends up being a dominant portion of the spend.

The way I see it, generally every new data point (on which the production model inference gets run once) becomes part of the data set which then gets used in training every next model, processing the same data point many more times in training, thus training unavoidably taking more effort than inference. Perhaps I'm a bit biased towards all kinds of self-supervised or human-in-the-loop or semi-supervised models, but…

Yup, exactly. It's a good point that for self-supervised workloads, the training set can become arbitrarily large. For a lot of other workloads in the vision space, most data needs to be labeled to be able to used for training.

Re: Are GPUs Worth It for ML?

#56

Post by jldugger » Mon, Aug 29, 2022, 10:13 PM UTC

PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

> If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

Wouldn't that depend on the size of your customer base? Or at least, requests per second?

Re: Are GPUs Worth It for ML?

#57

Post by synergy20 » Mon, Aug 29, 2022, 10:14 PM UTC

I think TPU is the way to go for ML, be it training or inference.

We're using GPU(some contains a TPU block inside) due to 'historical reasons'. With vector unit(x86 AVX, ARM SVE, RISC-V RVV) that is part of the host cpu, either put a TPU on a separate die of the chiplet, or just put it into a PCIe card will do the heavy lift ML job fine. It shall be much cheaper than the GPU model for ML nowadays, unless you are both a PC game player and a ML engineer.

Re: Are GPUs Worth It for ML?

#58

Post by triknomeister » Mon, Aug 29, 2022, 10:15 PM UTC

I thought this post would be about how ASICs are probably a better bet.

Re: Are GPUs Worth It for ML?

#59

Post by karamanolev » Mon, Aug 29, 2022, 10:15 PM UTC

jldugger wrote: post #56

PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

> If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference. Wouldn't that depend on the size of your customer base? Or at least, requests per second?

With more customers usually the revenue and profit grow, then the team becomes larger, wants to perform more experiments, spends more on training and so on. Inference is just so computationally cheap compared to training.

That's what I've seen in my experience, but I concur that there might be cases where the ML is a more-or-less solved problem for a very large customer base where inference is more. I've rarely seen it happen, but other people are sharing scenarios where it happens frequently. So I guess it massively depends on the domain.

Re: Are GPUs Worth It for ML?

#60

Post by version_five » Mon, Aug 29, 2022, 10:19 PM UTC

PeterisP wrote: post #45

varunkmohan wrote: post #12

Earlier quoted context omitted.

Agreed that there are workloads where inference is not expensive, but it's really workload dependent. For applications that run inference over large amounts of data in the computer vision space, inference ends up being a dominant portion of the spend.

The way I see it, generally every new data point (on which the production model inference gets run once) becomes part of the data set which then gets used in training every next model, processing the same data point many more times in training, thus training unavoidably taking more effort than inference. Perhaps I'm a bit biased towards all kinds of self-supervised or human-in-the-loop or semi-supervised models, but…

What you say re saving all data is the ideal. I'd add a couple caveats, one is that in many fields you often get lots of redundant data that adds nothing to training (for example if an image classifier looking for some rare class you can be drowning in images of the majority class). Or you can just have lots of data that is unambiguously and correctly classified- some kind of active learning can tell you what is worth keeping.

The other is that for various reasons the customer doesnt want to share their data (or at least have sharing built into the inference system) so even if you'd like to have everything they record, it's just not available. Obviously something to discourage but it seems common