Disappointed that there wasn’t anything on inference performance in the article at all. That’s what the major customers have announced they use it for.
CUDA Moat Still Alive
71–80 of 176 posts
Re: CUDA Moat Still Alive
#72> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.
This seems so insane, is anyone actually doing the work to provide an alternative to CUDA? Maybe Google?
Honestly, probably NVIDIA itself, since they contribute significantly to many open-source projects (MLIR), and also make their SoTA GEMM/Conv implementations open-source and available for study (Cutlass).
Re: CUDA Moat Still Alive
#73So basically Nvidia is like Windows and AMD is like Wine. I think trying to emulate CUDA and using forked Nvidia libraries is not the best strategy for AMD. They should have made a clean break and come out with a fresh API, like Apple's Metal.
Re: CUDA Moat Still Alive
#74AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain.
They don’t fucking want to! Believing this is anything like a market is fucking religion.
Re: CUDA Moat Still Alive
#75[flagged]
Lisa Su is an exemplary CEO, and widely recognized as such. She is exemplary for doing what she did with AMD, and did it without appealing at all to her sex... just on sheer competence. I think it's a bit presumptuous to suddenly call out her sex as if it matters. In reality, she's being talked about exactly like any male CEO. I have great faith in her though. She is clearly extraordinarily capable, and honestly a real inspiration to women in tech
Re: CUDA Moat Still Alive
#76Earlier quoted context omitted.
Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.
By overhead I'm talking about the things that have to be done supplementary to the algorithm. While there are complex state-of-the-art algorithms, those algorithms exist for everyone. The overhead is the bit that had to be done to make the algorithm work. For instance for sorting a list of strings the algorithm might be quick sort. The overhead would be in the efficiency of your string compare. For matmul I'm not sur…
> For matmul I'm not sure what your overhead is beyond moving memory, multiplying, and adding. A platform touting a memory bandwidth and raw compute advantage should have that covered. Where is the performance being lost?
The use of the word 'algorithm' is incorrect.
Look... I do this sort of work for a living. There has been no useful significant change to matmul algorithms.
What has changed is the matmul process.
Modern perf optimization on GPUs has little to do with algorithms and everything to do with process optimization. This is akin to factory floor planning and such. You have to make sure the data is there when the processing units need it, and the data is coming in at the fastest rate possible, while keeping everything synchronized to avoid wrong results or deadlocks.
Really compute power has nothing to do with it. It's a waste of time to even consider it. We can compute matmuls much faster than you can naively bring memory to the processing units. Whoever solves that problem will become very rich.
To that end, NVIDIA ships libraries that will choose from a wide variety of implementations the appropriate trade-offs necessary for SoTA perf on matmuls of all shapes and data types.
Re: CUDA Moat Still Alive
#77> CUDA Moat Still Alive Wrong conclusion. AMD is slower than NVidia, but not _that_ much slower. They are actually pretty cost-competitive. The just need to do some improvements, and they'll be a very viable competitor.
Just like that? So a little work and now they are competitive. You know how much work “just a little bit of work” is doing? They us a cultural issue and it will take months to fix if they are lucky and then you start tackling the tech debt they’ve built up. By that time it will be another generation
Re: CUDA Moat Still Alive
#78Earlier quoted context omitted.
The amount of effort this team took, literally co-opting AMD engineers, and working for 5 months, to get closer but not yet usable, means they are not even close to usable. What team wanting to do ML training/inference can afford so much down time for zero benefit? How many except a few big ones can get AMD to devote so many resources simply for that team? And, if you’re training a model costing you millions, the las…
Sure. But this work is done, and can be reused by others. Meanwhile, Nvidia hardware is expensive and still is in short supply. AMD might look quite tempting.
Like the article say, if the model change a little this work need to be almost thrown out
Re: CUDA Moat Still Alive
#79My anecdata on AMD hiring: they just aren't moving fast enough. They still wanted to fly people out scheduling 3 weeks in advance for AI compiler work. That's just not going to work. Startups and companies like NVIDIA, OpenAI are hiring much faster with much less onerous interview processes, with higher compensation. This is not a mystery. People work for money and aren't going to hop through more hoops to be paid less.
Re: CUDA Moat Still Alive
#80Earlier quoted context omitted.
> she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. It's not a deflection, but a straightforward description of AMDs current top-down market strategy of partnering with big players instead of doubling down to have a great OOBE for consumers & others who don't order GPUs by the pallet. It's an honest reflection if their current core comp…
That's the excuse used by every big company shitting out software so broken that it needs intensive professional babysitting. I've been on both sides of this shitshow, I've even said those lines before! But I've also been in the trenches making the broken shit work and I know that it's fundamentally an excuse. There's a reason why people pay 80% margin to Nvidia and there's a reason why AMD is worth less than the rou…
What exactly are they in denial about? They are aware that software is not a strength of theirs, so they partner with those who are great at it.
Would you say AMD is "shitting the bed" by not building it's own consoles too? You know AMD could build a kick-ass console since they are doing the heavy-lifting for the Playstation, and the XBox[1] , but AMD knows as much as anybody that they don't have the skills to wrangle studio relationships or figure out which games to finance. Instead, they lean hard in their HW skills and get Sony Entertainment/the Xbox division do what they do best.
1.and the Steam Deck, plus half a dozen Deck clones.