Are GPUs Worth It for ML? • Hacker News Board

Re: Are GPUs Worth It for ML?

#71

Post by [deleted] » Tue, Aug 30, 2022, 12:39 AM UTC

machinekob wrote: post #36

beecafe wrote: post #25

Earlier quoted context omitted.

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Ahh yes cause there is only one way to do Deep Learning and it is ofc stacking models large enough to not be useful outside pods of GPUs and this is for sure way to go if you want to make money (from VC ofc cause you wont have much users that are ever willing to pay so much that you'll ever make even, as was OpenAI and other big model providers, maybe you can get some money/sponsoring from state or uni). Market for l…

[deleted]

Re: Are GPUs Worth It for ML?

#72

Post by macksd » Tue, Aug 30, 2022, 12:40 AM UTC

beecafe wrote: post #25

machinekob wrote: post #17

Earlier quoted context omitted.

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easi…

You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.

Isn't a key selling point of the latest, hottest model that's on the front page of Hacker News multiple times right now, the fact that it fits on consumer-grade GPUs? Surely some of the interesting ideas it's spawning right now are people doing transfer learning on GPUs that don't end in "100", don't you think?

Re: Are GPUs Worth It for ML?

#73

Post by varunkmohan » Tue, Aug 30, 2022, 1:02 AM UTC

fancyfredbot wrote: post #61

varunkmohan wrote: post #39

Earlier quoted context omitted.

rCUDA is super cool! One of the issues though is for a lot of the common model frameworks are not supported and a new release has not come out a while.

Fair point. It's not obvious from the website which model frameworks does exafunction supports, or when the last exafunction release was.

Yeah, we should have a public release very soon for people to deploy internally. We will have support for all the commonly-used frameworks and different versions.

Re: Are GPUs Worth It for ML?

#74

Post by synergy20 » Tue, Aug 30, 2022, 1:19 AM UTC

why_only_15 wrote: post #70

synergy20 wrote: post #57

I think TPU is the way to go for ML, be it training or inference. We're using GPU(some contains a TPU block inside) due to 'historical reasons'. With vector unit(x86 AVX, ARM SVE, RISC-V RVV) that is part of the host cpu, either put a TPU on a separate die of the chiplet, or just put it into a PCIe card will do the heavy lift ML job fine. It shall be much cheaper than the GPU model for ML nowadays, unless you are bot…

It's true that we were initially using GPUs mostly for historical reasons, but over the last several years modern GPUs have been optimized for ML as much as anything else. If you read NVidia's marketing documents, they talk constantly about ML. The A100 is about as good, if not better, than the TPUv4 in terms of raw performance on ML workloads. The A100 can do 312 bf16 TFLOPs and costs $0.88/hr on Google Cloud [0] wh…

Maybe Google is favoring TPUv4 over whatever GPU runs on its platform?

With Hopper 100 on the way, I wonder when TPUv5 will come out.

I also wonder how Intel's Gaudi2 vs Ponte Vecchio will work together, looks like duplicate efforts for me.

AMD has its MI300 on the way, but it seems still far behind Nvidia|TPU|Intel at this point.

Re: Are GPUs Worth It for ML?

#75

Post by nl » Tue, Aug 30, 2022, 2:14 AM UTC

tombert wrote: post #10

37ef_ced3 wrote: post #4

For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/ Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com You can expect about 2x the performance of TensorFlow or PyTorc…

Is there a thing that Fabrice Bellard hasn't built? I had no idea that he was interested in something like machine learning, but I guess I shouldn't have been surprised because he has built every tool that I use.

If you are in the "data compression ~= intelligence" camp then Fabrice Bellard is currently leading the race to AI too.

http://prize.hutter1.net/

https://bellard.org/nncp/

http://www.mattmahoney.net/dc/text.html

Re: Are GPUs Worth It for ML?

#76

Post by fomine3 » Tue, Aug 30, 2022, 2:53 AM UTC

pqn wrote: post #6

Kukumber wrote: post #3

An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment

Disclaimer: I work at Exafunction I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand. But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.

GCP still provides NVIDIA K80. I wonder is it still worth to hold.

Re: Are GPUs Worth It for ML?

#77

Post by fomine3 » Tue, Aug 30, 2022, 2:57 AM UTC

einpoklum wrote: post #51

> And CPUs are so much cheaper Doesn't look like it. Consumer: AMD ThreadRipper 3970X: ~3000 USD on NewEgg https://www.newegg.com/amd-ryzen-threadripper-2990wx/p/N82E1... NVIDIA RTX 3080 Ti Founders' Edition: ~2000 USD https://www.newegg.com/nvidia-900-1g133-2518-000/p/1FT-0004-... For servers, a comparison is even more complicated and it wouldn't be fair to just give two numbers, but I still don't think GPUs are mor…

Consumer GPUs are very cheap but prohibited to use it on datacenter.

Re: Are GPUs Worth It for ML?

#78

Post by jimmygrapes » Tue, Aug 30, 2022, 3:12 AM UTC

Perhaps it's been mentioned before but I do find it curious how often crypto mining was lambasted for contributing to climate change get I haven't seen anybody bat an eye at a fairly similar amount of compute power used for ML applications. Makes me wonder.

Re: Are GPUs Worth It for ML?

#79

Post by gpm » Tue, Aug 30, 2022, 3:19 AM UTC

PeterisP wrote: post #7

For some reason they focus on the inference, which is the computationally cheap part. If you're working on ML (as opposed to deploying someone else's ML) then almost all of your workload is training, not inference.

Alpha zero used 5000 TPUs to generate games (inference only), and 16 to train the networks.

The split definitely depends on what you're doing past developing/deploying.

(Source: https://kstatic.googleusercontent.com/files/2f51b2a749a284c2...)

Re: Are GPUs Worth It for ML?

#80

Post by vaultminer » Tue, Aug 30, 2022, 3:42 AM UTC

synergy20 wrote: post #57

I think TPU is the way to go for ML, be it training or inference. We're using GPU(some contains a TPU block inside) due to 'historical reasons'. With vector unit(x86 AVX, ARM SVE, RISC-V RVV) that is part of the host cpu, either put a TPU on a separate die of the chiplet, or just put it into a PCIe card will do the heavy lift ML job fine. It shall be much cheaper than the GPU model for ML nowadays, unless you are bot…

No post body was provided.