Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA Moat Still Alive

semianalysis.com

Post reply on HN
Print view View original

91–100 of 176 posts

Re: CUDA Moat Still Alive

#91

Post by petters »

ryao wrote: post #89
kevingadd wrote: post #36

Earlier quoted context omitted.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

There are, but everyone uses variations of the same O(n^3) algorithm taught in college introduction to linear algebra classes because it is numerically stable and can be made extremely fast through tweaks that give spatial locality and good cache characteristics. Meanwhile the asymptomatically faster algorithms have such large constants in their big O notation that they are not worth using. FFT based matrix multiplic…

> matrix multiplication, which is O((n^2)log(n))

Isn't the fastest theoretical algorithm something like O(n^2.37) ?

Re: CUDA Moat Still Alive

#92

Post by cavisne »

I think it’s hardware not software.

“Cuda moat” is a misnomer. The PTX spec is relatively short (600 page pdf). Triton directly writes PTX, skipping cuda. Flash attention was created by a non nvidia employee without access to any of the secret sauce within Cuda or its libraries.

The hardware is just not as good, and no software can paper over its flaws.

Re: CUDA Moat Still Alive

#93

Post by jey »

petters wrote: post #91
ryao wrote: post #89

Earlier quoted context omitted.

There are, but everyone uses variations of the same O(n^3) algorithm taught in college introduction to linear algebra classes because it is numerically stable and can be made extremely fast through tweaks that give spatial locality and good cache characteristics. Meanwhile the asymptomatically faster algorithms have such large constants in their big O notation that they are not worth using. FFT based matrix multiplic…

> matrix multiplication, which is O((n^2)log(n)) Isn't the fastest theoretical algorithm something like O(n^2.37) ?

Yes, but it's impractical unless you have galactic-scale matrices to multiply (at least).

Re: CUDA Moat Still Alive

#94

Post by cavisne »

jiggawatts wrote: post #64
kevingadd wrote: post #36

Earlier quoted context omitted.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis. It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-som…

How do the cache sizes compare between AMD GPU’s and nvidia? I remember reading a while ago they were quite different (enough to make flash attention painful to implement)

Re: CUDA Moat Still Alive

#95

Post by croes »

ryao wrote: post #54

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: *…

You hardly beat someone by copying him. They have way more experience in the field you try to catch up.

Re: CUDA Moat Still Alive

#96

Post by atq2119 »

benreesman wrote: post #74

AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain. They don’t fucking want to! Believing this is anything like a market is fucking religion.

You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people.

Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high.

Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably some issues in upper management which could perhaps be fixed with some targeted hiring decisions, but do you really believe some random person on here would have a chance making that call?

Re: CUDA Moat Still Alive

#98

Post by eapriv »

ryao wrote: post #89
kevingadd wrote: post #36

Earlier quoted context omitted.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

There are, but everyone uses variations of the same O(n^3) algorithm taught in college introduction to linear algebra classes because it is numerically stable and can be made extremely fast through tweaks that give spatial locality and good cache characteristics. Meanwhile the asymptomatically faster algorithms have such large constants in their big O notation that they are not worth using. FFT based matrix multiplic…

> FFT based matrix multiplication, which is O((n^2)log(n))

What?

Re: CUDA Moat Still Alive

#99

Post by rahkiin »

AlotOfReading wrote: post #37
nrp wrote: post #13

> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.

Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.

That’s why they cooperate closely with imec and their FAB in Leuven

Re: CUDA Moat Still Alive

#100

Post by benreesman »

atq2119 wrote: post #96
benreesman wrote: post #74

AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain. They don’t fucking want to! Believing this is anything like a market is fucking religion.

You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people. Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high. Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably som…

We were arguing about this two years ago, maybe five. I was sharing NVIDIA dev boxes with other hackers doing CUDA in 2016.

There’s this meme that it can’t change on a dime and I believe that.

You could build this from scratch in a decade. JFK sent NASA to the moon in less time for comparable money.

If NVIDIA shareholders can’t come close? What fucking good are they? Why do our carrier battle groups guard their supply chain?

Post reply on HN