Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA Moat Still Alive

semianalysis.com

Post reply on HN
Print view View original

131–140 of 176 posts

Re: CUDA Moat Still Alive

#131

Post by ryao »

eapriv wrote: post #98
ryao wrote: post #89

Earlier quoted context omitted.

There are, but everyone uses variations of the same O(n^3) algorithm taught in college introduction to linear algebra classes because it is numerically stable and can be made extremely fast through tweaks that give spatial locality and good cache characteristics. Meanwhile the asymptomatically faster algorithms have such large constants in their big O notation that they are not worth using. FFT based matrix multiplic…

> FFT based matrix multiplication, which is O((n^2)log(n)) What?

https://en.wikipedia.org/wiki/Schönhage–Strassen_algorithm

I forgot the log(log(n)) factor.

In any case, for matrix multiplications that people actually do, this algorithm runs slower than a well optimized O(n^3) matrix multiplication implementation because the constant factor in the Big O notation is orders of magnitude larger.

Re: CUDA Moat Still Alive

#132

Post by ryao »

dogma1138 wrote: post #60
ryao wrote: post #47

Earlier quoted context omitted.

> GPU support lagged behind for years, no support for APUs and no guaranteed forward compatibility were clear signs that as a whole they have no idea what they are doing when it comes to building and shipping a software ecosystem. This is likely self inflicted. They decided to make two different architectures. One is CDNA for HPC and the other is RDNA for graphics. They are reportedly going to rectify this with UDNA…

This was happening before CDNA was even a thing. They didn’t release support even for all GPUs from the same generation and dropped support for GPUs sometime within 6 months of releasing a version that actually “worked”. The entire core architecture behind ROCM is rotten. P.S. NVIDIA usually has multiple CUDA feature levels even within a generation. The difference is that a) they always provide a fallback option, and…

The differences between CUDA feature levels appear minor according to the PTX documentation:

https://docs.nvidia.com/cuda/parallel-thread-execution/index...

They also appear to be cululative.

Re: CUDA Moat Still Alive

#133

Post by AlotOfReading »

Flenkno wrote: post #127
AlotOfReading wrote: post #37

Earlier quoted context omitted.

Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.

You are comparing a machine the size of a container with GPUs? its nice to be aware of this but this is so fastly different from a critisism point of view that i don't think that matters.

It's not a criticism, it's an extreme example from a company people know that I have no particular NDA restrictions with.

Re: CUDA Moat Still Alive

#134

Post by wjnc »

Lerc wrote: post #35
Archit3ch wrote: post #27

Earlier quoted context omitted.

Coming up next: "We bought AMD stock on the open market and used it to compensate AMD engineers".

You joke, but it is almost a genuine investment opportunity here for a large player. Spend a billion on AMD shares, Spend another Billion on a out-of-house software team to solve the software solution to more than double the share price. Taking into account that there are players that already own billions in AMD shares, they could probably do that as well. On the other hand perhaps it would be better for them, as maj…

I enjoy this train of thought a lot. Capturing shareholder value by just creating it yourself. The destruction of a moat I fear is worth less than the existence of a moat a competitor has successfully built. So don’t forget to buying puts Nvidia.

Re: CUDA Moat Still Alive

#135

Post by ryao »

Flenkno wrote: post #128
ryao wrote: post #54

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: *…

Nvidia has a unique problem, wants to move fast and has a shit load of money. No need for Nvidia to go first to an industry standard and neither for AMD. Personally would be great its getting backported but its so far away from an normal use case.

Nvidia is the one who went to an industry standard before their competitors in this space. It was created in 1999 and is called infiniband:

https://en.wikipedia.org/wiki/InfiniBand

Infiniband is extremely popular in the HPC space, which is why Nvidia adopted it. Everyone else saw Nvidia adopt it and said "Let us make a new network standard to be incompatible". This is mind boggling.

Even more mind boggling is that many of the companies in the Ultra Ethernet Consortium are members of the Infiniband Trade Association, AMD included:

https://www.infinibandta.org/member-listing/

This would be like the automotive industry forming a consortium to invent new incompatible wheels to exclude a successful upstart that adopted their existing standard wheel designs. With trillions of dollars in revenue on the line, you would think that companies would use existing networking standards to focus on building competitive hardware with reduced time to market, yet they are instead reinventing networking standards just because they can. This is a huge gift to Nvidia, since it means that everyone else is wasting time and money instead of being competitive.

Re: CUDA Moat Still Alive

#136

Post by sokoloff »

imtringued wrote: post #121
croes wrote: post #95

Earlier quoted context omitted.

You hardly beat someone by copying him. They have way more experience in the field you try to catch up.

AMD doesn't need to beat Nvidia, they just need to match them at a lower price point.

In business, that combination is nearly impossible to distinguish from beating them.

Re: CUDA Moat Still Alive

#137

Post by ryao »

jiggawatts wrote: post #107
Fronzie wrote: post #103

Earlier quoted context omitted.

> From what I've seen, AMD has done... none of this. There are a number of pull-requests to ROCMblas for tuning various sizes of GEMV and GEMM operations. For example: https://github.com/ROCm/rocBLAS/pull/1532

Merged two days ago!? That’s about half a decade after they should have done this foundational work! I guess it’s better late than never, but in this case a timely implementation was worth about a trillion dollars… maybe two.

There are likely plenty of unrealized opportunities to improve mature BLAS libraries. For example, this guy who was able to outperform OpenBLAS' GEMM on Zen 4:

https://salykova.github.io/matmul-cpu

Concidentally, the Intel MKL also outperforms OpenBLAS, so there being room for improvement is well known. That said, I have a GEMV implementation that outperforms both the Intel MKL and OpenBLAS in my tests on Zen 3:

https://github.com/ryao/llama3.c/blob/master/run.c#L429

That is unless you shoehorn GEMV into the Intel MKL's batched GEMM function, which then outperforms it when there is locality. Of course, when there is no locality, my code runs faster.

I suspect if/when this reaches the established amd64 BLAS implementations' authors, they will adopt my trick to get their non-batched GEMV implementations to run fast too. In particular, I am calculating the dot products for 8 rows in parallel followed by 8 parallel horizontal additions. I have not seen the 8 parallel horizontal addition technique mentioned anywhere, so I might be the first to have done it.

Re: CUDA Moat Still Alive

#138

Post by llamaimperative »

Flenkno wrote: post #127
AlotOfReading wrote: post #37

Earlier quoted context omitted.

Sadly common at hardware companies. The most extreme case I've heard of is ASML, who supposedly doesn't keep any machines of their own. They test against "almost-ready" machines right before they go out the door to customers.

You are comparing a machine the size of a container with GPUs? its nice to be aware of this but this is so fastly different from a critisism point of view that i don't think that matters.

I found it interesting ¯\_(ツ)_/¯

Re: CUDA Moat Still Alive

#139

Post by theptip »

nrp wrote: post #13

> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.

This is baffling. I’m sure there are many technical reasons I don’t grok that AMD’s job is challenging, but it’s wild that they are dropping the ball on such obvious stuff as this.

The prize is trillions of dollars, and they can print hundreds of millions if they can convince the market that they are closing the gap.

It’s embarrassing that whoever actually tries to use their product hits these crass bugs (same with geohot who was really invested in making AMD’s cards work; I think he just ran their demo script in a loop and produced crashes).

It seems they really don’t understand/value the developer flywheel.

Re: CUDA Moat Still Alive

#140

Post by mrlongroots »

ryao wrote: post #54

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: *…

> Infiniband is an industry standard

Infiniband is not an industry standard lol.

Maybe it used to be, but it definitely is not anymore. Most Infiniband vendors are dead. The only product from those days that endures is Cornelis' Omnipath, and even that only emulated the Infiniband API back with its first gen, and then evolved to be its own thing.

At this point, Infiniband is as good as a proprietary interconnect only sold by Nvidia/Mellanox.

Post reply on HN