Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making AMD GPUs competitive for LLM inference (2023)

blog.mlc.ai

Post reply on HN
Print view View original

141–150 of 221 posts

Re: Making AMD GPUs competitive for LLM inference (2023)

#141

Post by cameron_b »

claytonjy wrote: post #73
xedrac wrote: post #65

Earlier quoted context omitted.

Maybe the situation has gotten better in recent years, but my experience with Nvidia toolchains was a complete nightmare back in 2018.

The cuda situation is definitely better. The nvidia struggles are now with the higher-level software they’re pushing (triton, tensor-llm, riva, etc), tools that are the most performant option when they work, but a garbage developer experience when you step outside the golden path

I want to double-down on this statement, and call attention to the competitive nature of it. Specifically, I have recently tried to set up Triton on arm hardware. One might presume Nvidia would give attention to an architecture they develop, but the way forward is not easy. For some version of Ubuntu, you might have the correct version of python ( usually older than packaged ) but current LTS is out of luck for guidance or packages.

https://github.com/triton-lang/triton/issues/4978

Re: Making AMD GPUs competitive for LLM inference (2023)

#142

Post by steeve »

sebmellen wrote: post #21
jroesch wrote: post #11

Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.

Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.

We (ZML) have AMD MI300X working just fine, in fact, faster than H100

Re: Making AMD GPUs competitive for LLM inference (2023)

#143

Post by AnthonyMouse »

wqaatwt wrote: post #104
AnthonyMouse wrote: post #87

Earlier quoted context omitted.

Flash was popular because it was an attractive platform for the developer. Back then there was no HTML5 and browsers didn't otherwise support a lot of the things Flash did. Flash Player was an abomination, it was crashy and full of security vulnerabilities, but that was a problem for the user rather than the developer and it was the developer choosing what to use to make the site. This is pretty much exactly what hap…

> users have to use expensive hardware with proprietary drivers/firmware What do you mean by that? People trying to run their own models are not “the users” they are a tiny insignificant niche segment.

There are far more people running llama.cpp, various image generators, etc. than there are people developing that code. Even when the "users" are corporate entities, they're not necessarily doing any development in excess of integrating the existing code with their other systems.

We're also likely to see a stronger swing away from "do inference in the cloud" because of the aligned incentives of "companies don't want to pay for all that hardware and electricity" and "users have privacy concerns" such that companies doing inference on the local device will have both lower costs and a feature they can advertise over the competition.

What this is waiting for is hardware in the hands of the users that can actually do this for a mass market price, but there is no shortage of companies wanting a piece of that. In particular, Apple is going to be pushing that hard and despite the price they do a lot of volume, and then you're going to start seeing more PCs with high-VRAM GPUs or iGPUs with dedicated GDDR/HBM on the package as their competitors want feature parity for the thing everybody is talking about, the cost of which isn't actually that high, e.g. 40GB of GDDR6 is less than $100.

Re: Making AMD GPUs competitive for LLM inference (2023)

#144

Post by nine_k »

crest wrote: post #138
highwaylights wrote: post #132

Earlier quoted context omitted.

How does the compute performance compare to 4090’s for these workloads? (I release it will be significantly lower, just try to get as much of a comparison as is possible).

The Radeon VII is special compared to most older (and current) affordable GPUs in that it used HBM giving it memory bandwidth comparable to modern cards ~1TB/s and has reasonable FP64 (1:4) throughput instead of (1:64). So this card can still be pretty interesting for running memory bandwidth intensive FP64 workloads. Anything affordable afterward by either AMD or Nvidia crippled realistic FP64 throughput to below wh…

If we speak about FP64, are your loads more like fluid dynamics than ML training?

Re: Making AMD GPUs competitive for LLM inference (2023)

#145

Post by adrian_b »

llm_trw wrote: post #23
sebmellen wrote: post #21

Earlier quoted context omitted.

Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.

Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.

In CPUs, AMD has made many innovations that have been copied by Intel only after many years and this delay had an important contribution to Intel's downfall.

The most important has been the fact that AMD has predicted correctly that big monolithic CPUs will no longer be feasible in the future CMOS fabrication technologies, so they have designed the Zen family since the beginning with a chiplet-based architecture. Intel had attempted to ridicule them, but after losing many billions they have been forced to copy this strategy.

Also in the microarchitecture of their CPUs AMD has made the right choices since the beginning and then they have improved it constantly with each generation. The result is that now the latest Intel big core, Lion Cove, has a microarchitecture that is much more similar to AMD Zen 5 than to any of the previous Intel cores, because they had to do this to get a competitive core.

In the distant past, AMD has also introduced a lot of innovations long before they were copied by Intel, but it is true that those had not been invented by AMD, but they had been copied by AMD from more expensive CPUs, like DEC Alpha or Cray or IBM POWER, but Intel has also copied them only after being forced by the competition with AMD.

Re: Making AMD GPUs competitive for LLM inference (2023)

#146

Post by 7speter »

kimixa wrote: post #101
rockskon wrote: post #71

Earlier quoted context omitted.

They've actually been making positive moves with GPUs lately along with a success story for the B580.

B580 being a "success" is purely a business decision as a loss leader to get their name into the market. A larger die on a newer node than either Nvidia or AMD means their per-unit costs are higher, and are selling it at a lower price. That's not a long-term success strategy. Maybe good for getting your name in the conversation, but not sustainable.

I don’t know if this matters but while the B580 has a die comparable in size to a 4070 (~280mm^2), it has about half the transistors (~17-18 billion), iirc.

Re: Making AMD GPUs competitive for LLM inference (2023)

#147

Post by 7speter »

KeplerBoy wrote: post #122
oofabz wrote: post #86

Earlier quoted context omitted.

The die size of the B580 is 272 mm2, which is a lot of silicon for $249. The performance of the GPU is good for its price but bad for its die size. Manufacturing cost is closely tied to die size. 272 mm2 puts the B580 in the same league as the Radeon 7700XT, a $449 card, and the GeForce 4070 Super, which is $599. The idea that Intel is selling these cards at a loss sounds reasonable to me.

At a loss seems a bit overly dramatic. I'd guess Nvidia sells SKUs for three times their marginal cost. Intel is probably operating at cost without any hopes of recouping R&D with the current SKUs, but that's reasonable for an aspiring competitor.

It kinda seems they are covering the cost of throwing massive amounts of resources trying to get Arc’s drivers in shape.

Re: Making AMD GPUs competitive for LLM inference (2023)

#148

Post by Dylan16807 »

schmidtleonard wrote: post #128
Dylan16807 wrote: post #79

Earlier quoted context omitted.

> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link So you have a full terabyte per second of bandwidth? What GPU is that? (The 64GB/s number is an x4 link. If you meant you have over four times that, then it sounds like CXL would be pretty competitive.)

https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 Memory Size: 24 GB Memory Type: GDDR6X Memory Bus: 384 bit Bandwidth: 1.01 TB/s Bandwidth between where the LLM is stored and where your matrix*vector multiplies are done is the important figure for inference. You want to measure this in terabytes per second, not gigabytes per second. A 7900XTX also has 1TB/s on paper, but you'll need awkward workarounds ev…

A 4090 is not "years old pleb tier". Same for 3090 and 7900XTX.

There's a serious gap between CXL and RAM, but it's not nearly as big as it used to be.

Re: Making AMD GPUs competitive for LLM inference (2023)

#149

Post by Dylan16807 »

adrian_b wrote: post #140
Dylan16807 wrote: post #79

Earlier quoted context omitted.

> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link So you have a full terabyte per second of bandwidth? What GPU is that? (The 64GB/s number is an x4 link. If you meant you have over four times that, then it sounds like CXL would be pretty competitive.)

Already an ancient Radeon VII from 5 years ago had 1 terabyte per second of memory bandwidth. Later consumer GPUs have regressed and only RTX 4090 offers the same memory bandwidth in the current NVIDIA generation.

Radeon VII had HBM.

So I can understand a call for returning to HBM, but it's an expensive choice and doesn't fit the description.

Re: Making AMD GPUs competitive for LLM inference (2023)

#150

Post by AnthonyMouse »

llm_trw wrote: post #96
AnthonyMouse wrote: post #85

Earlier quoted context omitted.

> If I can save someone else from wasting $100,000 on hardware and six months of their life then my post has done more good than the AMD marketing department ever will. This seems like unuseful advice if you've already given up on them. You tried it and at some point in the past it wasn't ready. But by not being ready they're losing money, so they have a direct incentive to fix it. Which would take a certain amount o…

I've tried it three times. I've seen people try it every six months for two decades now. At some point you just have to accept that AMD is not a serious company, but is a second rate copycat and there is no way to change that without firing everyone from middle management up. I'm deeply worried about stagnation in the CPU space now that they are top dog and Intel is dead in the water. Here's hoping China and Risk V s…

> At some point you just have to accept that AMD is not a serious company, but is a second rate copycat and there is no way to change that without firing everyone from middle management up.

AMD has always punched above their weight. Historically their problem was that they were the much smaller company and under heavy resource constraints.

Around the turn of the century the Athlon was faster than the Pentium III and then they made x86 64-bit when Intel was trying to screw everyone with Itanic. But the Pentium 4 was a marketing-optimized design that maximized clock speed at the expense of heat and performance per clock. Intel was outselling them even though the Athlon 64 was at least as good if not better. The Pentium 4 was rubbish for laptops because of the heat problems, so Intel eventually had to design a separate chip for that, but they also had the resources to do it.

That was the point that AMD made their biggest mistake. When they set out to design their next chip the competition was the Pentium 4, so they made a power-hungry monster designed to hit high clock speeds at the expense of performance per clock. But the reason more people didn't buy the Athlon 64 wasn't that they couldn't figure out that a 2.4GHz CPU could be faster than a 2.8GHz CPU, it was all the anti-competitive shenanigans Intel was doing behind closed doors to e.g. keep PC OEMs from featuring systems with AMD CPUs. Meanwhile by then Intel had figured out that the Pentium 4 was, in fact, a bad design, when their own Pentium M laptops started outperforming the Pentium 4 desktops. So the Pentium 4 line got canceled and Bulldozer had to go up against the Pentium M-based Core, which nearly bankrupted AMD and compromised their ability to fund the R&D needed to sustain state of the art fabs.

Since then they've been climbing back out of the hole but it wasn't until Ryzen in 2017 that you could safely conclude they weren't on the verge of bankruptcy, and even then they were saddled with a lot of debt and contracts requiring them to use the uncompetitive Global Foundries fabs for several years. It wasn't until Zen4 in 2022 that they finally got to switch the whole package to TSMC.

So until quite recently the answer to the question "why didn't they do X?" was obvious. They didn't have the money. But now they do.

Post reply on HN