Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD may get across the CUDA moat

hpcwire.com

Post reply on HN
Print view View original

111–120 of 312 posts

Re: AMD may get across the CUDA moat

#111

Post by zirgs »

tyre wrote: post #82
binarymax wrote: post #11

Earlier quoted context omitted.

My intuition is along the lines that if AMD had a competing product earlier, then it would have kept prices down. But since Nvidia has shown what the market will pay, AMD won't be able to resist overcharging. It will probably come down a little, but nowhere near to the point of affordability. I sure hope I'm wrong.

AMD might have to charge less to break into customers that are already bought into Nvidia. There has to be a discount to cover the switching costs + still provide savings (or access).

AMD will have to provide a REALLY steep discount to convince me to come back.

Re: AMD may get across the CUDA moat

#112

Post by ris »

I don't understand the author's argument (if there is one) - pytorch has existed for ages. AMD's Instinct MI* range has existed for years now. If these are the key ingredients why has it not already happened?

Re: AMD may get across the CUDA moat

#113

Post by HideousKojima »

jjoonathan wrote: post #109
PH95VuimJjqBqy wrote: post #63

Earlier quoted context omitted.

I see these complains from time to time and I never understand them. I've literally been running nvidia on linux since the TNT2 days and have _never_ had this sort of issue. That's across many drivers and many cards over the many many years.

Same but linux experience is a steep and bumpy function of hardware. My guess: something like laptop GPU switching failed badly in the nvidia binary, earning it a reputation.

That was my experience, Nvidia Optimus (which is what allows dynamic switching between the integrated and dedicated GPU in laptops) was completely broken (as in a black screen, not just crashes or other issues) for several years, and Nvidia didn't care to do anything about it.

Re: AMD may get across the CUDA moat

#114

Post by lhl »

Havoc wrote: post #57
dylan604 wrote: post #29

Earlier quoted context omitted.

>If you can field a competitively priced consumer card if this unicorn were to show up, what's to say that all the non-consumers won't just scarf up these equally performant yet lower priced cards causing the supply-demand situation we're in now? the only difference would be a sudden supply of the expensive Nvidia cards that nobody wants because of their price.

>if this unicorn were to show up A unicorn like that showed up a couple hours ago. Someone posted a guide for getting llama to run on a 7900xtx https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_i... It's still slow and janky but this really isn't that far away. I don't buy that AMD can't make this happen if they actually tried. Go on fiverr, get them to compile a list of top 100 people in the DIY LLM space,…

llama.cpp, ExLlama, and MLC LLM have all had ROCm inferencing for months (here are a bunch of setup instructions I've written up, for Linux and Windows: https://llm-tracker.info/books/howto-guides/page/amd-gpus ) - but I don't think that's the problem (and wouldn't drive lots of volume or having downstream impact in any case).

The bigger problem is on the training/research support. Eg, here's no official support for AMD GPUs for bitsandbytes, and no support at all for FlashAttention/FA2 (nothing that 100K in hardware/grants to Dettmers or Dao's labs wouldn't fix I suspect).

The real elephant though is that AMD still having the disconnect that lack of support for consumer cards and home/academic devs in general has been disastrous (while Nvidia supports CUDA on basically every single GPU they've made since 2010) - just last week there was this mindblowing thread where it turns out an AMD employee is paying out of pocket for AMD GPUs to support build/CI for drivers on Debian. I mean, WTF, that's stupidity that's beyond embarrassing and gets into negligence terriroty IMO: https://news.ycombinator.com/item?id=37665784

Re: AMD may get across the CUDA moat

#115

Post by amelius »

mikepurvis wrote: post #67
omneity wrote: post #52

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than o…

Sigh. It's great that these container images exist to give people an easy on-ramp, but they definitely don't work for every use case (especially once you're in embedded where space matters and you might not be online to pull multi-gb updates from some registry). So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image the…

> So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day.

Sadly if e.g. 95% of their users can use the container, then it could make economical sense to do it that way.

Re: AMD may get across the CUDA moat

#116

Post by PH95VuimJjqBqy »

HideousKojima wrote: post #113
jjoonathan wrote: post #109

Earlier quoted context omitted.

Same but linux experience is a steep and bumpy function of hardware. My guess: something like laptop GPU switching failed badly in the nvidia binary, earning it a reputation.

That was my experience, Nvidia Optimus (which is what allows dynamic switching between the integrated and dedicated GPU in laptops) was completely broken (as in a black screen, not just crashes or other issues) for several years, and Nvidia didn't care to do anything about it.

I don't run laptops except when work requires it and that tends to be windows so that may explain the difference in experience.

Re: AMD may get across the CUDA moat

#117

Post by PH95VuimJjqBqy »

temp0826 wrote: post #94
PH95VuimJjqBqy wrote: post #63

Earlier quoted context omitted.

I see these complains from time to time and I never understand them. I've literally been running nvidia on linux since the TNT2 days and have _never_ had this sort of issue. That's across many drivers and many cards over the many many years.

I understand it, but I also haven't had any trouble since I figured out the right procedure for me on fedora (which probably took some time, but it's been so long that I can't remember). Whenever I read people having issues it sounds like they are using a package installed via dnf for the driver/etc. I've always had issues with dkms and the like and just install the latest .run from nvidia's website whenever I have a…

actually, it's a good point because that's how I always install nvidia drivers as well. Never from the local package manager.

Re: AMD may get across the CUDA moat

#118

Post by PH95VuimJjqBqy »

ant6n wrote: post #64
PH95VuimJjqBqy wrote: post #63

Earlier quoted context omitted.

I see these complains from time to time and I never understand them. I've literally been running nvidia on linux since the TNT2 days and have _never_ had this sort of issue. That's across many drivers and many cards over the many many years.

Well tnt2 should be pretty well supported by now ;-)

lmao, touche :)

Re: AMD may get across the CUDA moat

#119

Post by lhl »

HideousKojima wrote: post #113
jjoonathan wrote: post #109

Earlier quoted context omitted.

Same but linux experience is a steep and bumpy function of hardware. My guess: something like laptop GPU switching failed badly in the nvidia binary, earning it a reputation.

That was my experience, Nvidia Optimus (which is what allows dynamic switching between the integrated and dedicated GPU in laptops) was completely broken (as in a black screen, not just crashes or other issues) for several years, and Nvidia didn't care to do anything about it.

Yeah, Optimus was a huge PITA. I remember fighting with workarounds like bumblebee and prime for years. Also Nvidia dragged their feet on Wayland support for a few years too (and simultaneously was seemingly intent on sabotaging Nouveau).

Re: AMD may get across the CUDA moat

#120

Post by fotcorn »

pama wrote: post #72

There is only limited empirical evidence of AMD closing the gap that NVidia has created in the science or ML software. Even when considering pytorch only, the engineering effort to maintain specialized ROCm along with CUDA solutions is not trivial (think flashattention, or any customization that optimizes your own model). If your GPUs only need a simple ML workflow all times for a few years nonstop, maybe there exist…

ROCm has HIP (1) which is a compatibility layer to run CUDA code on AMD GPUs. In theory, you only have to adjust #includes, and everything should just work, but as usual, reality is different.

Newer backends for AI frameworks like OpenXLA and OpenAI Triton directly generate GPU native code using MLIR and LLVM, they do not use CUDA apart from some glue code to actually load the code onto the GPU and get the data there. Both already support ROCm, but from what I've read the support is not as mature yet compared to NVIDIA.

1: https://github.com/ROCm-Developer-Tools/HIP

Post reply on HN