Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD may get across the CUDA moat

hpcwire.com

Post reply on HN
Print view View original

261–270 of 312 posts

Re: AMD may get across the CUDA moat

#261

Post by kkielhofner »

mr_toad wrote: post #203
kkielhofner wrote: post #187

Earlier quoted context omitted.

Describing kernel panics and general nightmare scenarios as the general course with Nvidia doesn’t make sense either. Nvidia has 80% market share of the discrete GPU desktop market and at least 90% market share of cloud/datacenter. Nvidia GPUs are used almost exclusively for every cloud powered AI service and to train virtually every ML model in existence. Almost always on Linux. Do you really think any of this would…

People use flakey software all their time. As long as it mostly works most of the time most people put up with it. Examples: Windows in the 90’s and 00’s, or any AAA game on first release in the last 10 years.

I have a friend at the Facebook AI Research lab and I assure you they would not tolerate any level of fundamental flakiness from their 8,000 GPU cluster. Talent, opportunity cost, and time to market in general is so crucial in AI no one has any time or patience for the "oddball Linux desktop" experiences people are describing here.

Gaming users may tolerate some flakiness for their hobby but these AI companies dealing in the nine-figure range (minimum) absolutely do not.

Re: AMD may get across the CUDA moat

#263

Post by falconroar »

pama wrote: post #72

There is only limited empirical evidence of AMD closing the gap that NVidia has created in the science or ML software. Even when considering pytorch only, the engineering effort to maintain specialized ROCm along with CUDA solutions is not trivial (think flashattention, or any customization that optimizes your own model). If your GPUs only need a simple ML workflow all times for a few years nonstop, maybe there exist…

I don't understand why developers of PyTorch and similar don't use OpenCL. Open standard, runs everywhere, similar performance - what's the downside??

Re: AMD may get across the CUDA moat

#264

Post by LoganDark »

falconroar wrote: post #262

Is there any reason OpenCL is not the standard in implementations like PyTorch? Similar performance, open standard, runs everywhere - what's the downside?

IIRC, ease of implementation (for the GPU kernels), and cross-compatibility (the same bytecode can be loaded by multiple models of GPU).

Re: AMD may get across the CUDA moat

#265

Post by kkielhofner »

errnoh wrote: post #236
kkielhofner wrote: post #190

Earlier quoted context omitted.

Whisper is actually a great example of why Nvidia has such a stronghold on ML/AI and why it’s so difficult to compete. There’s getting something to “work”, which is often enough of a challenge with ROCm. Then there’s getting it to work well (next challenge). Then there’s getting it to work as well as Nvidia/CUDA. With Whisper, as one example, you should be running it with ctranslate2[0]. Of all the platforms on their…

While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant". https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080,…

> While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant".

It easily is. See the benchmarks[0] from faster-whisper which uses Ctranslate2. That's 5x faster than OpenAI reference code on a Tesla V100. Needless to say something like a 4080 easily multiplies that.

> https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant.

With all due respect to the author of the article this is "my first entry into ML" territory. They talk about a 5-10 second delay, my project can do sub 1 second times[1] even with ancient GPUs thanks to Ctranslate2. I don't have an RTX 4080 but if you look at the performance stats for the closest thing (RTX 4090) the performance numbers are positively bonkers - completely untouchable for anything ROCm based. Same goes for the other projects I linked, lmdeploy does over 100 tokens/s in a single session with LLama2 13b on my RTX 4090 and almost 600 tokens/s across eight simultaneous sessions.

> EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )

I don't understand what you're saying here. It (along with the other projects I linked here[2]) are fantastic examples of just how far behind the ROCm ecosystem is. ROCm isn't even on the radar for most of them as your linked issue highlights.

Things always get implemented in CUDA first (ten years in this space and I've never seen ROCm first) and ROCm users either wait months (minimum) for sub-par performance or never get it at all.

[0] - https://github.com/guillaumekln/faster-whisper#benchmark

[1] - https://heywillow.io/components/willow-inference-server/#ben...

[2] - https://news.ycombinator.com/item?id=37793635#37798902

Re: AMD may get across the CUDA moat

#266

Post by raggi »

mschuetz wrote: post #229
raggi wrote: post #148

Can we just get wgsl compute good enough and over the line instead, and do away with these moats?

Not happening. WGSL wants to support the lowest common denominator, so it'll always mainly be a 5-year old mobile-phone API. Also if you want to beat CUDA, you'll need some functionality that's completely missing in compute shaders, especially WGSL. Like pointers and pointer casting (and that glsl buffer reference extension is the worst emulation of that feature I've every seen).

The language extensions feature is designed to provide these kinds of facilities is it not?

Re: AMD may get across the CUDA moat

#267

Post by hot_gril »

65a wrote: post #202
hot_gril wrote: post #80

Earlier quoted context omitted.

I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs; I know they're not exactly the same instruction set as Intel, and more importantly they work with a different (and less mainstream) set of mobos, so I don't want em. If AMD manages to pull more customers their way, that's great, it just means lower Intel premium for me.

As an owner of some Sapphire Rapids parts, let me just direct you to: https://edc.intel.com/content/www/us/en/design/products-and-...

To see errata tracked by Intel is a good sign.

Re: AMD may get across the CUDA moat

#268

Post by hot_gril »

hot_gril wrote: post #80
fluxem wrote: post #65

I call it the 90% problem. If AMD works for 90% of my projects, I would still buy NVIDIA, which works for 100%, even though I’m paying a premium

I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs; I know they're not exactly the same instruction set as Intel, and more importantly they work with a different (and less mainstream) set of mobos, so I don't want em. If AMD manages to pull more customers their way, that's great, it just means lower Intel premium for me.

Forgot to also mention iGPU and other on-chip accelerators being different and Intel usually having the edge there.

Re: AMD may get across the CUDA moat

#269

Post by pama »

falconroar wrote: post #263
pama wrote: post #72

There is only limited empirical evidence of AMD closing the gap that NVidia has created in the science or ML software. Even when considering pytorch only, the engineering effort to maintain specialized ROCm along with CUDA solutions is not trivial (think flashattention, or any customization that optimizes your own model). If your GPUs only need a simple ML workflow all times for a few years nonstop, maybe there exist…

I don't understand why developers of PyTorch and similar don't use OpenCL. Open standard, runs everywhere, similar performance - what's the downside??

I don’t know for sure why the early pytorch team picked it, but my guess is due to simplicity and performance. NVidia optimizes CUDA better that OpenCL and provides tons of useful performance tuning tools. It is hard to match the CUDA performance with OpenCL even on the same NVidia GPU hardware, and making performant code compatible across different GPU with OpenCL is also hard. I know examples of scientific codes that became simpler and faster (on nvidia hardware) by going from openCL to CUDA but haven’t yet heard of examples the other way around.

Re: AMD may get across the CUDA moat

#270

Post by anon291 »

Blammar wrote: post #176
runiq wrote: post #160

Earlier quoted context omitted.

Yeah, that's a pretty shortsighted take of things. Do you really believe that Nvidia hasn't taken steps do make sure their moat is as wide as possible?

The thing about owning the CUDA spec is that Nvidia can add new features quickly without having to argue with other hardware vendors. I find that a positive thing overall. Also, I choose to pay the ~$120 Windows tax once (per box), everything works very well, and I don't have the driver issues that some fraction of other users seem to have with Linux and Nvidia cards. Seems like a good use of my time.

Literally never had an issue with Nvidia and Linux in decades. Despite this, my windows installs have all sorts of issues.. as always
Post reply on HN