61–70 of 100 posts
There are some pretty elegant solutions out there for the problem of having the right ratio of CPU to GPU. One of the nicer ones is rCUDA. https://scholar.google.com/citations?view_op=view_citation&h...
rCUDA is super cool! One of the issues though is for a lot of the common model frameworks are not supported and a new release has not come out a while.
Earlier quoted context omitted.
What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…
You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.
There is tons of value to be had from smaller models. Even some state of the art results can be obtained on a relatively small set of commodity GPUs. Not everything is GPT-scale.
An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment
Disclaimer: I work at Exafunction I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand. But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.
Earlier quoted context omitted.
the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here
In what context? HPC and certain code bases have been effectively leveraging heterogenous CPU GPU workloads for a variety of applications for quite awhile. I know of some doing so in at least 2009 and know plenty of prior art was already there by that point, it's just a specific time I happen to remember.
reading again - it seems this paper calls HPC with GPUs a slightly different name "GPGPU" and lists the research activity separately.. so I didn't see it as HPC; basically what I wrote is not accurate. got it
Earlier quoted context omitted.
Disclaimer: I'm the Cofounder / CEO at Exafunction That's a great point. We'll be addressing this in an upcoming post as well. We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for…
Disclaimer != Disclosure Probably one of HNs most common mistakes in comments
For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.
Maybe from the researcher or data scientist's perspective. But if you have a product that uses ML and inference doesn't dominate training, you're doing it wrong.
If it doesn't work it has to be retrained on new data again and there are no efficient alternatives to this energy waste other than use more GPUs, TPUs, etc emitting more CO2 after years of Deep Learning existing.
A complete waste of resources and energy. Therefore it is not worth it at all.
I think TPU is the way to go for ML, be it training or inference. We're using GPU(some contains a TPU block inside) due to 'historical reasons'. With vector unit(x86 AVX, ARM SVE, RISC-V RVV) that is part of the host cpu, either put a TPU on a separate die of the chiplet, or just put it into a PCIe card will do the heavy lift ML job fine. It shall be much cheaper than the GPU model for ML nowadays, unless you are bot…
WRT putting a TPU on a separate die -- this has been done for several years in the mobile space: Apple Neural Engine for iPhones, TPU (not same as server TPU) on Pixel, SNPE on Qualcomm, etc.
[0] https://cloud.google.com/compute/gpus-pricing
[1] https://cloud.google.com/tpu/pricing#v4-pricing
[2] this is somewhat unfair, because the GPU pricing number is for just the GPU and not the host it runs on, whereas the TPU pricing number (for TPU VMs) includes the host it runs on. If you include the price GCP charges for the host, preemptible A100s are about $1.20/hr. Why does Google make GPUs look cheaper than TPUs when they're not? Your guess is as good as mine.