Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA Moat Still Alive

semianalysis.com

Post reply on HN
Print view View original

161–170 of 176 posts

Re: CUDA Moat Still Alive

#161

Post by ryao »

imtringued wrote: post #158
ryao wrote: post #90

Earlier quoted context omitted.

To be fair, GEMV is memory bandwidth bound and that is what token generation in transformers uses. GEMM is the compute bound one, provided you do not shoehorn GEMV into it. That special case is memory bandwidth bound.

GEMM isn't compute bound in ML in practice. If you do naive GEMM based attention, then you will have to write the output matrix into HBM and in the worst case you might even have to reload the output from HBM! So what is done in practice is an algorithm that doesn't calculate the same result, but is imperceptibly close to doing classic attention, namely flash attention. Flash attention lets you fuse the kernel so tha…

I have been working on inference software for my own use, with both CPU and GPU versions:

https://github.com/ryao/llama3.c

The only thing you wrote that makes any sense to me is “Flash attention lets you fuse the kernel”. Everything else you wrote makes no sense to me. For what it is worth, flash attention does not apply to llama 3 inference as far as I can tell.

Re: CUDA Moat Still Alive

#162

Post by pella »

pella wrote: post #155

Latest: Dylan Patel ( SemiAnalysis ) "Met with @LisaSu today for 1.5 hours as we went through everything She acknowledged the gaps in AMD software stack She took our specific recommendations seriously She asked her team and us a lot of questions Many changes are in flight already! Excited to see improvements coming" https://x.com/dylan522p/status/1871287937268383867

Lisa Su : https://x.com/LisaSu/status/1871362304194859511

"Thanks @dylan522p for the constructive conversation today. Feedback is a gift even when it’s critical. We have put a ton of work into customer and workload optimizations but there is lots more we can do to enable the broad ecosystem. I appreciate all the feedback and desire to engage with @AMD. We are committed to building a world-class open software stack. Lots planned for 2025. Happy holidays to all!"

Re: CUDA Moat Still Alive

#163

Post by mrlongroots »

ryao wrote: post #153
mrlongroots wrote: post #143

Earlier quoted context omitted.

Sorry to hound you for the third time, but this is wrong: > Infiniband is extremely popular in the HPC space Not anymore. There used to be Cray Aries/GNI, psm/psm2, and now there's Slingshot, the new Cornelis stuff etc. There's almost no Infiniband now.

The top500 says otherwise: https://www.infinibandta.org/infiniband-and-roce-advances-fu... Where are you getting your information?

I concede that the specifics of what I said were wrong, but the larger point was not.

If you buy a single DGX H100 rack and run LINPACK, you automatically get TOP500-grade numbers. Infiniband is a solid product, if not the best commercial offering for AI/ML, but no one buys it for an HPC cluster separately from the DGX boxes.

Re: CUDA Moat Still Alive

#164

Post by mrlongroots »

ryao wrote: post #146
mrlongroots wrote: post #144

Earlier quoted context omitted.

> Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. This is wrong. RDMA over Ethernet is... RDMA over Ethernet. There is no Infiniband involved. RoCE was motivated by supporting RDMA, which was then an IB-only feature, over regular Ethernet. The user-level APIs are the same (verbs), but the underlying architecture…

This is not what I have been told by others.

Tip: copy-paste our entire exchange as a Gemini prompt and ask "who's correct?".

(I did that yesterday and was very satisfied :) )

Re: CUDA Moat Still Alive

#165

Post by pjmlp »

Comma2976 wrote: post #147
pjmlp wrote: post #122

Earlier quoted context omitted.

As means to avoid paying for Windows licenses. All the games that matter are Windows games running via Proton, as Valve has failed to actually build a GNU/Linux native games ecosystem, in spite of UNIX/POSIX underpinnings of Android NDK, PlayStation, the studios hardly bother. The day Microsoft actually decides to challenge Proton, or do a netbooks move on handhelds with XBox OS/Windows, the SteamDeck will lose, just…

As a means of control

If they cared about control, they wouldn't depend on Windows ecosystem, rather foster GNU/Linux native games.

Everyone is quite curious what Microsoft will drop at CES 2025, and which OEMs will be on their side, it is going to be netbooks all over again.

Re: CUDA Moat Still Alive

#166

Post by mike_hearn »

sangnoir wrote: post #157
mike_hearn wrote: post #151

Earlier quoted context omitted.

It's in the article. Meta don't use AMD for training and write their own kernels for inference. You can't train with AMD, full stop, because their software stack is so buggy.

> It's in the article The same article also states that AMD provided custom bug-fixes written by Principle Engineers to address bugs in a benchmark - this is software that will only become part of the public release in 2 quarters. I ask again, do you think AMD will not expedite non-public bug-fixes for hyperscalers? > You can't train with AMD, full stop, because their software stack is so buggy. Point 7 from the arti…

AI labs don't want to train models using a stack build some guy hacked up on his desktop last night that's been through no proper QA process. The cost of a training run that fails or results in a garbage model due to numerical errors are huge.

Which is why, as they say clearly, nobody is training models on AMD. Only inference, at most. I'm not sure why you keep claiming they are training using private drivers. They clearly aren't.

Re: CUDA Moat Still Alive

#167

Post by javier2 »

sangnoir wrote: post #62
schmidtleonard wrote: post #59

Earlier quoted context omitted.

I don't have the inside baseball but I have seen those weird as hell interviews with Lisa Su where she gets asked point blank about the software problems and instead of "working on it, stay tuned" -- an answer that costs nothing to give -- she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. No, the horsepower of your F1 racecar doesn't ma…

> she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. It's not a deflection, but a straightforward description of AMDs current top-down market strategy of partnering with big players instead of doubling down to have a great OOBE for consumers & others who don't order GPUs by the pallet. It's an honest reflection if their current core comp…

The ones buying gpus in pallets _really_ need the software to not be a worry. Software issues would ruin the entire operational life

Re: CUDA Moat Still Alive

#168

Post by Comma2976 »

pjmlp wrote: post #165
Comma2976 wrote: post #147

Earlier quoted context omitted.

As a means of control

If they cared about control, they wouldn't depend on Windows ecosystem, rather foster GNU/Linux native games. Everyone is quite curious what Microsoft will drop at CES 2025, and which OEMs will be on their side, it is going to be netbooks all over again.

They are literally fostering Linux games by selling and endorsing a platform where those games would be native as well as having native releases of their own games. They aren't gonna force any third-party devs to do the same, but they're showing that there is a market while also growing it.

Re: CUDA Moat Still Alive

#169

Post by ryao »

mrlongroots wrote: post #163
ryao wrote: post #153

Earlier quoted context omitted.

The top500 says otherwise: https://www.infinibandta.org/infiniband-and-roce-advances-fu... Where are you getting your information?

I concede that the specifics of what I said were wrong, but the larger point was not. If you buy a single DGX H100 rack and run LINPACK, you automatically get TOP500-grade numbers. Infiniband is a solid product, if not the best commercial offering for AI/ML, but no one buys it for an HPC cluster separately from the DGX boxes.

#26 on the list uses AMD GPUs with infiniband:

https://www.top500.org/system/180171/

You can likely find more. Infiniband has been excellent for HPC since the 2000s. That includes all HPC workloads, not just AI/ML.

Excuse me if I do not believe your claims concerning infiniband. They contradict not only actual data, but also what I have heard from people I consider experts.

Also, you did not answer my question concerning the origin of your information. I notice from another comment if yours that you have been talking to a LLM about this conversation. Have you been posting things that a LLM tells you?

Re: CUDA Moat Still Alive

#170

Post by ryao »

mrlongroots wrote: post #164
ryao wrote: post #146

Earlier quoted context omitted.

This is not what I have been told by others.

Tip: copy-paste our entire exchange as a Gemini prompt and ask "who's correct?". (I did that yesterday and was very satisfied :) )

You are not the first person to say “the AI assistant liked what I said over what you said” to me on the Internet, to which I say they will tell you whatever you want to hear.

I am not interested in continuing this discussion, but if you want to do your own research, I suggest starting with the fact that the Infiniband Trade Association controls the RoCE specification. I suggest you avoid using LLMs, for obvious reasons.

Post reply on HN