hopper (H100) is the predecessor to the current blackwell architecture. This is a new AMD vs last generation nvidia benchmark.
Blackwell won't be here till next year.
AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
21–30 of 273 posts
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#22I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.
It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.
The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will.
While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy.
Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) - but how much of that current value is real rather than an AI speculation bubble?
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#23Cool story. How supported is OpenCL compared to CUDA again?
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#24Earlier quoted context omitted.
Pretty much and the test suit is optimized to get the results they wanted.
Pretty sure a useful benchmark for this kind of thing would calculate performance per watt (or per watt and dollar). That info is conspicuously absent from the article.
The electricity consumption in the cloud is not really important.
The H100 rents for about $4.5/hr consuming 0.7kWh in that hour which will likely cost them less than 7 cents.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#25I'm skeptical of these benchmarks for a number of reasons.
1. They're only comparing against VLLM, which isn't SOTA for latency-focused inference. For example, their vllm benchmark on 2 GPUs sees 102 tokens/s for BS=1, gpt-fast gets around 190 tok/s. https://github.com/pytorch-labs/gpt-fast 2. As others have pointed out, they're comparing H100 running with TP=2 vs. 2 AMD GPUs running independently.
Specifically,
> To make an accurate comparison between the systems with different settings of tensor parallelism, we extrapolate throughput for the MI300X by 2.
This is uhh.... very misleading, for a number of reasons. For one, at BS=1, what does running with 2 GPUs even mean? Do they mean that they're getting the results for one AMD GPUs at BS=1 and then... doubling that? Isn't that just... running at BS=2?
3. It's very strange to me that their throughput nearly doubles going from BS=1 to BS=2. MoE models have an interesting property that low amounts of batching doesn't actually significantly improve their throughput, and so on their Nvidia vllm benchmark they just go from 102 => 105 tokens/s throughput when going from BS=1 to BS=2. But on AMD GPUs they go from 142 to 280? That doesn't make any sense to me.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#26Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#27> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…
It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#28At this point in history were still at ROCm vs CUDA... Schmicko hardware is only as good as the software you can write for it.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#29I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.
We are in the middle of an LLM bubble.
Nvidia problem will sort itself out naturally in the coming months/years.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#30I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.
It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.
[deleted]