Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alternative(s) to run CUDA on non-Nvidia hardware

hpcwire.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#51

Post by superkuh »

woctordho wrote: post #13

There's nothing wrong to run CUDA on non-Nvidia hardware. CUDA has an interface that is reasonably well-designed, well-documented/reverse-engineered, and battle-tested for decades. What we need is not to invent another interface just under the name of 'open standard', but to implement the same interface. ROCm is exactly doing this, and so are other hardware SDKs such as MooreThread and Alibaba T-Head.

The difference between ROCm and CUDA is that when a consumer GPU is released by nvidia it's supported for CUDA for about a decade (1xxx series cards just dropped last year). When a consumer GPU is released by AMD it's not supported by ROCm till about a year after release and then it's supported for about 3-4 years. With the RX 580 there were only 3.7 years after release before ROCm support was pulled. I bought mine a couple years after release and so only had about a year and a half of ROCm. Never again.

Things might be different in enterprise but for consumer AMD GPU ROCm is a trap. It is a mayfly. Sure, you can try to run the cards unsupported but you're just multiplying the difficulty and maintainence burden. And nothing will just work.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#52

Post by xyzsparetimexyz »

pjmlp wrote: post #27
mschuetz wrote: post #20

Earlier quoted context omitted.

A couple of years ago I evaluated both Vulkan and Cuda as a choice for future projects. I couldnt get anything done after a week in Vulkan, but had the test prototype project working after just a day in Cuda. Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.

I used to be big into Khronos API camp, even did my project thesis in OpenGL, up to the famous Long Peaks fail. Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia). The "we build the specification", and then "t…

If you think that Vulkan is extension spaghetti you're clearly using it wrong. Set the API to 1.4 and many existing extensions get merged in.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#53

Post by xyzsparetimexyz »

mschuetz wrote: post #37
DiabloD3 wrote: post #29

Earlier quoted context omitted.

Weird, most people have the exact opposite experience. Having to deal with closed source opaque poorly documented stacks sucks.

They really don't, no. Vulkan: 50 lines to allocate device memory. Cuda: One single line. What kind of extensive documentation stack do you want for functionality that is trivial in Cuda? And that exact issue continues through every little step of the way to your first usable application. I know there is VMA, it is a very poor solution to a problem that shouldn't even exist, and it only poorly addresses one of 100 pa…

It's quite easy to set up a light abstraction layer with Vulkan where you simply use VMA, buffer device addresses and push constants for everything. No descriptor sets or bindings anything.

Alternatively you can use one of many abstraction layers that do this for you.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#54

Post by pjmlp »

xyzsparetimexyz wrote: post #52
pjmlp wrote: post #27

Earlier quoted context omitted.

I used to be big into Khronos API camp, even did my project thesis in OpenGL, up to the famous Long Peaks fail. Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia). The "we build the specification", and then "t…

If you think that Vulkan is extension spaghetti you're clearly using it wrong. Set the API to 1.4 and many existing extensions get merged in.

If you think changing to Vulkan 1.4 solves all the problems, you clearly aren't writing cross platform code.

First of all, that isn't even a thing if you need to target Android, or embedded hardware, secondly there are other extensions on the horizon.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#55

Post by ychen306 »

msond wrote: post #3
pjmlp wrote: post #2

Most of these "alternatives" focus on CUDA C++, and overlook what actually makes CUDA interesting. Already in 2020, https://developer.nvidia.com/blog/cuda-refresher-the-gpu-com...

We're actually targeting all of it, and not just CUDA C++.

How do you deal with target-specific inline asm like tcgen05.mma?

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#56

Post by msond »

HarHarVeryFunny wrote: post #50

These efforts to support CUDA on non-Nvidia hardware seem to me misguided. If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that (and torch.compile uses Triton for target-specific optimization). OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUD…

We actually support NVIDIA hardware, too.

In some benchmarks, SCALE beats nvcc, and we have compiler optimizations in the pipeline that will improve those numbers over time.

> If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that

Somewhat true, but, CUDA is significantly larger than PyTorch and there's more to Accelerated Computing than just those types of applications supported there.

> OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUDA translation layer on non-NVidia hardware would be an even worse idea.

SOTA mlperf submissions use CUDA to achieve their high levels of performance.

It's not a "translation layer", it's a native, ahead-of-time compiler that makes full use of the native hardware features. Here's an example of a feature (Shuffles) being compiled to take advantage of native hardware instructions, resulting in speedups: https://scale-lang.com/posts/2026-01-19-optimizing-cuda-shuf...

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#57

Post by HelloNurse »

DiabloD3 wrote: post #28
binsquare wrote: post #18

Earlier quoted context omitted.

Yep, there are inference stacks where it just does not work without cuda in any meaningful performance

Weird, since the most used open source inference engine is faster on Vulkan on platforms that offer multiple options, with the sole exception being Nvidia, due to poor Nvidia driver quality (which I am forced to assume is intentional, Nvidia wishes to maintain their moat after all).

Being fast and being as easy to program as CUDA are two different things.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#58

Post by mschuetz »

HarHarVeryFunny wrote: post #50

These efforts to support CUDA on non-Nvidia hardware seem to me misguided. If all you want is to be able to easily use non-NVidia hardware then high level tools like PyTorch already let you do that (and torch.compile uses Triton for target-specific optimization). OTOH if you want to be programming close to the metal to achieve top performance then you are probably not using CUDA in the first place, and using some CUD…

On the contrary, it's great. Cuda is the single sane compute API and system, so I'll use it even if it means being vendor-locked. If my CUDA programs start running elsewhere without much intervention, that'd be amazing

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#59

Post by xyzsparetimexyz »

pjmlp wrote: post #54
xyzsparetimexyz wrote: post #52

Earlier quoted context omitted.

If you think that Vulkan is extension spaghetti you're clearly using it wrong. Set the API to 1.4 and many existing extensions get merged in.

If you think changing to Vulkan 1.4 solves all the problems, you clearly aren't writing cross platform code. First of all, that isn't even a thing if you need to target Android, or embedded hardware, secondly there are other extensions on the horizon.

The vast majority of vulkan usecases aren't android or embedded. I indeed wouldn't recommend it there.

Re: Alternative(s) to run CUDA on non-Nvidia hardware

#60

Post by mschuetz »

xyzsparetimexyz wrote: post #53
mschuetz wrote: post #37

Earlier quoted context omitted.

They really don't, no. Vulkan: 50 lines to allocate device memory. Cuda: One single line. What kind of extensive documentation stack do you want for functionality that is trivial in Cuda? And that exact issue continues through every little step of the way to your first usable application. I know there is VMA, it is a very poor solution to a problem that shouldn't even exist, and it only poorly addresses one of 100 pa…

It's quite easy to set up a light abstraction layer with Vulkan where you simply use VMA, buffer device addresses and push constants for everything. No descriptor sets or bindings anything. Alternatively you can use one of many abstraction layers that do this for you.

It absolutely isn't. After having spent 5 days not getting anything done in Vulkan, and being able to implement that same thing in a single day in Cuda (no prior experience in either API), I decided to never ever use Vulkan. It's a hopelessly overengineered API that is in dire need of a successor.

I may give it another try once it does not require a wrapper before it is remotely usable. I.e., once it has a single-line malloc without the need for third-party libs; default queues so I don't need to query and select queues; implicit sync by default and explicit sync by choice; NV-style bindless (i.e. no descriptors, just a handle); and so much more.

Post reply on HN