Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA Moat Still Alive

semianalysis.com

Post reply on HN
Print view View original

101–110 of 176 posts

Re: CUDA Moat Still Alive

#101

Post by benreesman »

benreesman wrote: post #100
atq2119 wrote: post #96

Earlier quoted context omitted.

You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people. Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high. Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably som…

We were arguing about this two years ago, maybe five. I was sharing NVIDIA dev boxes with other hackers doing CUDA in 2016. There’s this meme that it can’t change on a dime and I believe that. You could build this from scratch in a decade. JFK sent NASA to the moon in less time for comparable money. If NVIDIA shareholders can’t come close? What fucking good are they? Why do our carrier battle groups guard their suppl…

Everyone is so concerned about “losing” the “AI” “race” to the PRC.

I say ship them Altman and an exaflop and watch their society corrupt itself at a fractal nature at machine speed.

Good fucking riddance. I see your fentanyl crisis: raise you Sam and a failure to ship GPT-5. Have fun with that.

Re: CUDA Moat Still Alive

#102

Post by klelatti »

AMD doesn’t just have to fix these issues it has to build up a record of fixing issues like those discussed here.

Otherwise who will bet their firm / cash / career on new hardware without a successful track record.

Re: CUDA Moat Still Alive

#103

Post by Fronzie »

jiggawatts wrote: post #64
kevingadd wrote: post #36

Earlier quoted context omitted.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis. It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-som…

> From what I've seen, AMD has done... none of this.

There are a number of pull-requests to ROCMblas for tuning various sizes of GEMV and GEMM operations. For example: https://github.com/ROCm/rocBLAS/pull/1532

Re: CUDA Moat Still Alive

#104

Post by Fronzie »

ryao wrote: post #48
roenxi wrote: post #19

Earlier quoted context omitted.

I expect everyone has been saying it for a while, the calls are just getting more strident and public as it becomes clear that AMD's failures are strategic rather than tactical. And as people try to build business on their half-hearted attempts. I still think it is a mistake to say that CUDA is a moat. IMO the problem here is that AMD still doesn't seem to think that GPGPU compute is a thing. They don't seem to under…

I wonder to what extent vulkan compute could be used for this. Of course, it is only an option on their RDNA GPUs since CDNA is not for graphics, even though that is the G in GPU.

There has been some testing within llama.cpp, which supports both Vulkan and ROCM-Blas. When it works, the latter is about 2x faster than the Vulkan version.

Re: CUDA Moat Still Alive

#106

Post by whazor »

AlotOfReading wrote: post #37
nrp wrote: post #13

> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.

Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.

Actually, they shipped not-ready machines to customers. In hope they could find solutions that can fix the machine later.

Re: CUDA Moat Still Alive

#107

Post by jiggawatts »

Fronzie wrote: post #103
jiggawatts wrote: post #64

Earlier quoted context omitted.

Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis. It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-som…

> From what I've seen, AMD has done... none of this. There are a number of pull-requests to ROCMblas for tuning various sizes of GEMV and GEMM operations. For example: https://github.com/ROCm/rocBLAS/pull/1532

Merged two days ago!?

That’s about half a decade after they should have done this foundational work!

I guess it’s better late than never, but in this case a timely implementation was worth about a trillion dollars… maybe two.

Re: CUDA Moat Still Alive

#108

Post by adrian_b »

kevingadd wrote: post #36
Lerc wrote: post #32

That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation. I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

Low overhead in the sense that matrix multiplication is almost the only algorithm that is able to reach computational throughput values very close to the theoretical maximum for a given hardware.

Good CPUs and GPUs have a throughput in Flop/s for matrix multiplication that is between 60% and 90% of the maximum possible throughput, with many (especially the CPUs) reaching values towards the high end of that range.

As shown in the article, the AMD GPUs attain only slightly less than 50% (for BF16; for FP8 the AMD efficiency is even less than 40%).

Such a low efficiency for the most important operation is not acceptable.

Re: CUDA Moat Still Alive

#109

Post by jhanschoo »

eapriv wrote: post #98
ryao wrote: post #89

Earlier quoted context omitted.

There are, but everyone uses variations of the same O(n^3) algorithm taught in college introduction to linear algebra classes because it is numerically stable and can be made extremely fast through tweaks that give spatial locality and good cache characteristics. Meanwhile the asymptomatically faster algorithms have such large constants in their big O notation that they are not worth using. FFT based matrix multiplic…

> FFT based matrix multiplication, which is O((n^2)log(n)) What?

FFT is fast Fourier transform, and our best theoretical bounds on multiplication come from methods involving FFT.

Re: CUDA Moat Still Alive

#110

Post by pjmlp »

sangnoir wrote: post #80
schmidtleonard wrote: post #69

Earlier quoted context omitted.

That's the excuse used by every big company shitting out software so broken that it needs intensive professional babysitting. I've been on both sides of this shitshow, I've even said those lines before! But I've also been in the trenches making the broken shit work and I know that it's fundamentally an excuse. There's a reason why people pay 80% margin to Nvidia and there's a reason why AMD is worth less than the rou…

What exactly are they in denial about? They are aware that software is not a strength of theirs, so they partner with those who are great at it. Would you say AMD is "shitting the bed" by not building it's own consoles too? You know AMD could build a kick-ass console since they are doing the heavy-lifting for the Playstation, and the XBox[1] , but AMD knows as much as anybody that they don't have the skills to wrangl…

It would go nowhere, games history is full of great hardware that died because they failed to have a profitable ecosystem.

Even Steam Deck is only a success, because it depends on Windows ecosystem, and the moment Microsoft decides it is enough, lets see how long it holds.

Post reply on HN