An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment
*only in the cloud environment
Throw some 3090s in a rack and you’ll break even in 3 months
Are GPUs Worth It for ML?
11–20 of 100 posts
Re: Are GPUs Worth It for ML?
#12For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
Agreed that there are workloads where inference is not expensive, but it's really workload dependent. For applications that run inference over large amounts of data in the computer vision space, inference ends up being a dominant portion of the spend.
Re: Are GPUs Worth It for ML?
#13An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment
the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here
Re: Are GPUs Worth It for ML?
#14For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/ Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com You can expect about 2x the performance of TensorFlow or PyTorc…
Is there a thing that Fabrice Bellard hasn't built? I had no idea that he was interested in something like machine learning, but I guess I shouldn't have been surprised because he has built every tool that I use.
Re: Are GPUs Worth It for ML?
#15We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.…
Disclaimer: I'm the Cofounder / CEO at Exafunction
That's a great point. We'll be addressing this in an upcoming post as well.
We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for latency sensitive workloads, running on spot instances or CPUs sometimes is not an option.
I could definitely see cases where it makes sense to run on spot CPUs though.
Re: Are GPUs Worth It for ML?
#16For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/ Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com You can expect about 2x the performance of TensorFlow or PyTorc…
Is there a thing that Fabrice Bellard hasn't built? I had no idea that he was interested in something like machine learning, but I guess I shouldn't have been surprised because he has built every tool that I use.
[deleted]
Re: Are GPUs Worth It for ML?
#17We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.…
What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training?
Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090.
And if you are solo dev its even easier choice as you can reuse your rig for other stuff when you dont train anything (for example gaming :D).
Only possibility is if you get free 100k from AWS and then 100k from GCP you can live with that for a year or even two if u stack both providers but it is special case and im not sure how easy it is to get 100k right now.
Re: Are GPUs Worth It for ML?
#18For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
I have not found this to be true at all in my field (natural language generation).
We have a 7 figure GPU setup that is running 24/7 at 100% utilization just to handle inference.
Re: Are GPUs Worth It for ML?
#19For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
I have not found this to be true at all in my field (natural language generation). We have a 7 figure GPU setup that is running 24/7 at 100% utilization just to handle inference.
How do you train new models if your GPUs are being used for inference? I guess the training happens significantly less frequently?
Forgive my ignorance.
Re: Are GPUs Worth It for ML?
#20Earlier quoted context omitted.
I have not found this to be true at all in my field (natural language generation). We have a 7 figure GPU setup that is running 24/7 at 100% utilization just to handle inference.
How do you train new models if your GPUs are being used for inference? I guess the training happens significantly less frequently? Forgive my ignorance.
Typically a different set of hardware for model training.