Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD may get across the CUDA moat

hpcwire.com

Post reply on HN
Print view View original

191–200 of 312 posts

Re: AMD may get across the CUDA moat

#191

Post by nextaccountic »

javchz wrote: post #26

CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. Running Nvidia in Linux isn't as much fun. Fedora and Debian can be incredibly reliable systems, but when you add an Nvidia card, I feel like I am back in Windows Vista with kernel crashes from time to time.

> CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful.

What AMD really needs is to have 100% feature parity with CUDA without changing a single line of code. Maybe for this to happen it needs to add hardware features or something (I see people saying that CUDA as an API is very tailored to the capabilities of nvidia GPUs), I don't know.

If AMD relies on people changing their code to make it portable, it already lost.

Re: AMD may get across the CUDA moat

#192

Post by kkielhofner »

einpoklum wrote: post #137
nabla9 wrote: post #28

> Crossing the CUDA moat for AMD GPUs may be as easy as using PyTorch. Nvidia has spent huge amount of work to make code run smoothly and fast. AMD has to work hard to catch up. ROCm code is slower , has more bugs, don't have enough features and they have compatibility issues between cards.

> Nvidia has spent huge amount of work to make code run smoothly and fast. Well, let's say "smoother" rather than "smoothly". > ROCm code is slower On physically-comparable hardware? Possible, but that's not an easy claim to make, certainly not as expansively as you have. References? > has more bugs Possible, but - NVIDIA keeps their bug database secret. I'm guessing you're concluding this from anecdotal experience?…

I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References:

https://github.com/InternLM/lmdeploy

https://github.com/vllm-project/vllm

https://github.com/OpenNMT/CTranslate2

You know what’s missing from all of these and many more like them? Support for ROCm. This is all before you get to the really wildly performant stuff like Triton Inference Server, FasterTransformer, TensorRT-LLM, etc.

ROCm is at the “get it to work stage” (see top comment, blog posts everywhere celebrating minor successes, etc). CUDA is at the “wring every last penny of performance out of this thing” stage.

In terms of hardware support, I think that one is obvious. The U in CUDA originally stood for unified. Look at the list of chips supported by Nvidia drivers and CUDA releases. Literally anything from at least the past 10 years that has Nvidia printed on the box will just run CUDA code.

One of my projects specifically targets Pascal up - when I thought even Pascal was a stretch. Cue my surprise when I got a report of someone casually firing it up on Maxwell when I was pretty certain there was no way it could work.

A Maxwell laptop chip. It also runs just as well on an H100.

THAT is hardware support.

Re: AMD may get across the CUDA moat

#193

Post by mgaunard »

mikepurvis wrote: post #121
xahrepap wrote: post #105

Earlier quoted context omitted.

I know it's still different than what you're looking for, so you probably already know this, but many projects like this have the Dockerfile on github which shows exactly how they set up the image. For example: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master... They also have some for Fedora. Looks like for this you need to install their repo: curl -sL https://repo.radeon.com/rocm/rocm.gpg.key | apt-key…

Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "kno…

I don't know what world you live in, but this is a problem for any software development.

You need to ensure that there is only one version of any library used globally throughout the code and that the set of versions is compatible with each other, and preferably you also want everything to be built against the same toolchain with the same flags.

That usually means onboarding third-party libraries into your own build system.

Re: AMD may get across the CUDA moat

#194

Post by aiunboxed »

withwarmup wrote: post #145

CUDA is the result of years of NVIDIA supporting the ecosystem, some people likes to complain because they bought hardware that was cheaper but can't use it for what they want to use it, when you buy NVIDIA, you aren't buying only the hardware, but the insane amount of work they have put into the ecosystem, the same goes for Intel, mkl and scikit-learn intelex aren't free to develop. AMD has the hardware but the supp…

Exactly, with NVIDIAs core focus on AI way before it was cool has lead to them being in this advantageous position. For AMD just being a price friendly competitor to Intel and Nvidia was the motto.

Re: AMD may get across the CUDA moat

#195

Post by xcdzvyn »

latchkey wrote: post #100
freedomben wrote: post #96

Earlier quoted context omitted.

what type of cards did you have? what did you do with them after PoS? How did you even buy so many cards? Sorry, like the other commenter I'm extremely curious

Primarily 470,480,570,580. We also ran a very large cluster of PS5 APU chips too. Got the chips directly from AMD. Since these are 4-5 year old chips, they were not going to ever be used. It is more ROI efficient with ETH mining to use older cards than newer ones. Had a couple OEM manufacture the cards specially for us with 8gb, heatsinks instead of fans (lower power usage) and no display ports (lower cost). They wil…

Cool! Were the PS5 APUs actually attached to a PS5 motherboard, or were they repurposed entirely?

Re: AMD may get across the CUDA moat

#196

Post by latchkey »

xcdzvyn wrote: post #195
latchkey wrote: post #100

Earlier quoted context omitted.

Primarily 470,480,570,580. We also ran a very large cluster of PS5 APU chips too. Got the chips directly from AMD. Since these are 4-5 year old chips, they were not going to ever be used. It is more ROI efficient with ETH mining to use older cards than newer ones. Had a couple OEM manufacture the cards specially for us with 8gb, heatsinks instead of fans (lower power usage) and no display ports (lower cost). They wil…

Cool! Were the PS5 APUs actually attached to a PS5 motherboard, or were they repurposed entirely?

Asrock bc-250. This is some hardware that I wouldn't have purchased, if given the choice, especially that close to ETH PoS.

That said, I made it work, which was an insane amount of work, and it mined really well.

Re: AMD may get across the CUDA moat

#197

Post by pedrovhb »

RockRobotRock wrote: post #129
omneity wrote: post #52

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than o…

Have you gotten it to work with Whisper by any chance?

I've had luck with an RX5700XT and whisper.cpp built with clblast. Works like a charm, not entirely a scarring experience getting it to work (easier than most other stuff which was surprising to me).

One arcane detail is that whereas for PyTorch I have to set the env var HSA_OVERRIDE_GFX_VERSION to 10.3.0, getting it to run with whisper.cpp and llama.cpp requires setting it to 10.1.0. Good luck and may it cost you less hair than it did me.

Re: AMD may get across the CUDA moat

#198

Post by upbeat_general »

This article doesn’t address the real challenge [in my mind].

Framework support is one thing, but what about the million standalone CUDA kernels that have been written, especially common in research. Nobody wants to spend time re-writing/porting those, especially when they probably don’t understand the low-level details in the first place.

Not to mention, what is the plan for comprehensive framework support? I’ve experienced the pain of porting models to different hardware architectures where various ops are unsupported. Is it realistic to get full coverage of e.g., PyTorch?

Re: AMD may get across the CUDA moat

#199

Post by 65a »

treprinum wrote: post #90
SoftTalker wrote: post #61

Earlier quoted context omitted.

The main issue with Nvidia on Linux AIUI is that they don't release the source code for their drivers.

That might be a philosophical problem that never prevented me from training models on Linux. The half-baked half-crashing AMD solutions just lead to wasting time I can spend on ML research instead.

I literally gave away my last laptop with a discrete nVidia card because it wasted so much of my time.

Re: AMD may get across the CUDA moat

#200

Post by incognition »

omneity wrote: post #52

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than o…

Fp64??
Post reply on HN