Are GPUs Worth It for ML?
exafunction.com
Are GPUs Worth It for ML?
1–10 of 100 posts
Re: Are GPUs Worth It for ML?
#2[deleted]
Re: Are GPUs Worth It for ML?
#3An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment
Re: Are GPUs Worth It for ML?
#4For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/
Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com
You can expect about 2x the performance of TensorFlow or PyTorch.
Re: Are GPUs Worth It for ML?
#5[deleted]
Re: Are GPUs Worth It for ML?
#6An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment
Disclaimer: I work at Exafunction
I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand.
But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.
Re: Are GPUs Worth It for ML?
#7For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
Re: Are GPUs Worth It for ML?
#8It depends a lot on your problem, of course.
Game-playing (e.g. AlphaGo) is computationally hard but the rules are immutable, target functions (e.g., heuristics) don’t change much, and you can generate arbitrarily sized clean data sets (play more games). On these problems, ML-scaling approaches work very well. For business problems where the value of data decays rapidly, though, you probably don’t need the power of a deer or complex neural net with millions of parameters, and expensive specialty hardware probably isn’t worth it.
Re: Are GPUs Worth It for ML?
#9We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.freecodecamp.org/news/ml-armada-running-tens-of-...
Training was always GPUs (for speed), non-spot-instance (for reliability), and cloud based (for infinite parallelism). Training work tended to be chunky, never made sense to build servers in house that would be idle some of the time, and queued at other times.
Re: Are GPUs Worth It for ML?
#10For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/ Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com You can expect about 2x the performance of TensorFlow or PyTorc…
Is there a thing that Fabrice Bellard hasn't built? I had no idea that he was interested in something like machine learning, but I guess I shouldn't have been surprised because he has built every tool that I use.