Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA Moat Still Alive

semianalysis.com

Post reply on HN
Print view View original

61–70 of 176 posts

Re: CUDA Moat Still Alive

#61

Post by dogma1138 »

almostgotcaught wrote: post #42
dogma1138 wrote: post #20

Earlier quoted context omitted.

Anyone who looks at the mess that is ROCm and the design choices they made could easily see that. GPU support lagged behind for years, no support for APUs and no guaranteed forward compatibility were clear signs that as a whole they have no idea what they are doing when it comes to building and shipping a software ecosystem. To that you can add the long history of both AMD and ATI before they merged releasing dog shi…

> On the other hand you can take any CUDA binary even one that dates back to the original Tesla and run it on any modern NVIDIA GPU This particular difference stems the fact that NVIDIA has PTX and AMD does not have any such thing. Ie this kind of backwards compatibility will never be possible on AMD.

Backward compatibility is one thing but not having a forward compatibility is a killer.

Having to create a binary that targets a very specific set of hardware and having no guarantees and in fact having a guarantee that it won’t on future hardware is what make ROCM unusable for anything you intend to ship.

What’s worse is that they also drop support for their GPUs faster than Leo drops support for his girlfriends once they reach 25…

So not only that you have to recompile there is no guarantee that your code would work with future versions of ROCM or that future versions of ROCM could still produce binaries which are compatible with your older hardware.

Like how is this not the first design goal to address when you are building a CUDA competitor I don’t fucking know.

Re: CUDA Moat Still Alive

#62

Post by sangnoir »

schmidtleonard wrote: post #59
Lerc wrote: post #35

Earlier quoted context omitted.

You joke, but it is almost a genuine investment opportunity here for a large player. Spend a billion on AMD shares, Spend another Billion on a out-of-house software team to solve the software solution to more than double the share price. Taking into account that there are players that already own billions in AMD shares, they could probably do that as well. On the other hand perhaps it would be better for them, as maj…

I don't have the inside baseball but I have seen those weird as hell interviews with Lisa Su where she gets asked point blank about the software problems and instead of "working on it, stay tuned" -- an answer that costs nothing to give -- she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. No, the horsepower of your F1 racecar doesn't ma…

> she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having.

It's not a deflection, but a straightforward description of AMDs current top-down market strategy of partnering with big players instead of doubling down to have a great OOBE for consumers & others who don't order GPUs by the pallet. It's an honest reflection if their current core competencies, and the opportunity presented by Nvidia's margins.

They are going for a bang-for-buck right now aiming at data center workloads, and the hyperscalers care a lot about perf/$ than raw performance at. Hyperscalers are also more self-sufficient at software: they have entire teams working on PyTorch, Jax, and writing kernels.

Re: CUDA Moat Still Alive

#63

Post by almostgotcaught »

dogma1138 wrote: post #61
almostgotcaught wrote: post #42

Earlier quoted context omitted.

> On the other hand you can take any CUDA binary even one that dates back to the original Tesla and run it on any modern NVIDIA GPU This particular difference stems the fact that NVIDIA has PTX and AMD does not have any such thing. Ie this kind of backwards compatibility will never be possible on AMD.

Backward compatibility is one thing but not having a forward compatibility is a killer. Having to create a binary that targets a very specific set of hardware and having no guarantees and in fact having a guarantee that it won’t on future hardware is what make ROCM unusable for anything you intend to ship. What’s worse is that they also drop support for their GPUs faster than Leo drops support for his girlfriends onc…

> Like how is this not the first design goal to address when you are building a CUDA competitor I don’t fucking know.

The words "tech debt" do not have any meaning at AMD. No one understands why this is a problem.

Re: CUDA Moat Still Alive

#64

Post by jiggawatts »

kevingadd wrote: post #36
Lerc wrote: post #32

That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation. I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis.

It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-something-percent of the best available algorithms!

All NVIDIA did was "put the effort in", where the effort isn't some super clever algorithm implemented by a unique genius, but they simply made hundreds of variants of the matmul algorithm optimised for various scenarios. It's a kind of algorithmic brute force for eking out every last percentage point for every shape and size of input matrices on every GPU model and even for various SLI configurations.

From what I've seen, AMD has done... none of this.

Re: CUDA Moat Still Alive

#65

Post by adgjlsfhk1 »

dmoy wrote: post #39
AlotOfReading wrote: post #37

Earlier quoted context omitted.

Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.

ASML might be an extreme outlier though, don't those things cost like $50 million+ each?

try 400 mil

Re: CUDA Moat Still Alive

#66

Post by Lerc »

kevingadd wrote: post #36
Lerc wrote: post #32

That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation. I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

By overhead I'm talking about the things that have to be done supplementary to the algorithm.

While there are complex state-of-the-art algorithms, those algorithms exist for everyone. The overhead is the bit that had to be done to make the algorithm work.

For instance for sorting a list of strings the algorithm might be quick sort. The overhead would be in the efficiency of your string compare.

For matmul I'm not sure what your overhead is beyond moving memory, multiplying, and adding. A platform touting a memory bandwidth and raw compute advantage should have that covered. Where is the performance being lost?

I guess the only real options are stalls, unnecessary copies, or unnecessary computations.

Re: CUDA Moat Still Alive

#67

Post by bobmcnamara »

tonetegeatinst wrote: post #46
dmoy wrote: post #39

Earlier quoted context omitted.

ASML might be an extreme outlier though, don't those things cost like $50 million+ each?

Many for last gen process nodes, and from a second or third hand supplier if you could even find one. ASML makes very few fully working machines each year, and the cost and throughput those machines have is astronomical. They have spare parts you'd bet, and I'd bet they have some SLA agreement with each customer where an engineer is basically on call nearby in case a single thing dosnt work or a random part breaks or…

Almost a decade ago KFAB had a fire, power was cut, everything in process was dumped, they planned to restart but ended up being was cheaper to close the whole facility .

Probably for the best though, KFAB had been discharging several tons of solvents, cleaning agents, and reagents per year into the surrounding area [for as long as it ran](https://enviro.epa.gov/facts/tri/ef-facilities/#/Release/640...)

https://enviro.epa.gov/facts/tri/ef-facilities/#/Release/640...

Re: CUDA Moat Still Alive

#68

Post by dhruvdh »

Lerc wrote: post #66
kevingadd wrote: post #36

Earlier quoted context omitted.

Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.

By overhead I'm talking about the things that have to be done supplementary to the algorithm. While there are complex state-of-the-art algorithms, those algorithms exist for everyone. The overhead is the bit that had to be done to make the algorithm work. For instance for sorting a list of strings the algorithm might be quick sort. The overhead would be in the efficiency of your string compare. For matmul I'm not sur…

Which algorithm you pick for what shape of matrices is different and not straightforward to figure out. AMD currently wants you to “tune” ops and likely search for the right algorithm for your shapes while Nvidia has accurate heuristics for picking the right algorithm.

Re: CUDA Moat Still Alive

#69

Post by schmidtleonard »

sangnoir wrote: post #62
schmidtleonard wrote: post #59

Earlier quoted context omitted.

I don't have the inside baseball but I have seen those weird as hell interviews with Lisa Su where she gets asked point blank about the software problems and instead of "working on it, stay tuned" -- an answer that costs nothing to give -- she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. No, the horsepower of your F1 racecar doesn't ma…

> she deflects into "performance is what matters," which is the kind of denial that rhymes exactly with the problems they are having. It's not a deflection, but a straightforward description of AMDs current top-down market strategy of partnering with big players instead of doubling down to have a great OOBE for consumers & others who don't order GPUs by the pallet. It's an honest reflection if their current core comp…

That's the excuse used by every big company shitting out software so broken that it needs intensive professional babysitting.

I've been on both sides of this shitshow, I've even said those lines before! But I've also been in the trenches making the broken shit work and I know that it's fundamentally an excuse. There's a reason why people pay 80% margin to Nvidia and there's a reason why AMD is worth less than the rounding error when people call NVDA a 3 trillion dollar company.

It's not because people can't read a spec sheet, it's because people want their expensive engineers training models not changing diapers on incontinent equipment.

I hope AMD pulls through but denial is _not_ the move.

Re: CUDA Moat Still Alive

#70

Post by Implicated »

dzdt wrote: post #24

[flagged]

> Is it a kind of misogynism?

-.-

At what point did any of the criticism have anything to do with her gender? Honest question, I'm scratching my head trying to see where misogyny comes into play. Surely it's not that _because_ she's a woman any criticism from men must be misogynistic? Would it be different if Intel's CEO was female? Or do the people criticising need to be of the same gender as those they're criticising in order for there to be no misogyny?

Truly just trying to get an idea of what sort of perspective it takes to get to

> Is it a kind of misogynism?

Post reply on HN