Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

251–260 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#251

Post by sigmoid10 »

versteegen wrote: post #117
logicchains wrote: post #95

Earlier quoted context omitted.

That's a pretty big claim, that Microsoft and Meta have their own proprietary cuda-replacement stack. Do you have any evidence for that claim?

I'm guessing what they meant is that they use toolchains that are retargetable to other GPUs (and typically compile down to PTX (nVidia assembly language) on nVidia GPUs rather than go through CUDA source -- GCC and clang can both target PTX). For example XLA and most SYSCL toolchains support much more than nVidia.

Even then it's an insanely bold assumption that a company other than Nvidia could build a better framework than CUDA for compiling PTX. Especially since CUDA is already so much C-like. I've never seen anyone go deeper than that outside of academia.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#252

Post by muxr »

hackerlight wrote: post #184
ddtaylor wrote: post #161

Earlier quoted context omitted.

I love AMD, but my Nvidia stock position currently is much higher than AMD.

AMD is at a much higher PE ratio. Is the market expecting AMD to up its game in the GPU sector? Or is the market expecting a pullback in GPU demand due to possibility for non-GPU AI solutions becoming the frontier or for AI investment to slow down?

AMD doesn't have a higher PE ratio. You should use nonGAAP numbers because the GAAP numbers include Xilinx Googwill Amortization which skew the PE.

AMD's PE is ~55. Nvidia's PE is above 70.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#253

Post by muxr »

aurareturn wrote: post #18
impulser_ wrote: post #15

Earlier quoted context omitted.

If they used Nvidia's chip would this somehow make the blog post better?

For one, they didn't use TensorRT in the test. Also, stuff like this is hard to take the results seriously: * To make an accurate comparison between the systems with different settings of tensor parallelism, we extrapolate throughput for the MI300X by 2. * All inference frameworks are configured to use FP16 compute paths. Enabling FP8 compute is left for future work. They did everything they can to make sure AMD is f…

I don't understand why they should use TensorRT. vLLM is much more popular and it was actually written for Nvidia. It also supports AMD hardware, so it's the appropriate tool to compare.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#254

Post by muxr »

imtringued wrote: post #221
mistymountains wrote: post #207

Earlier quoted context omitted.

These kinds of comments make me think few people have actually tried. My experience has been 1 work day of getting things set up to work the same as before for training and testing (PyTorch).

You have to consider that the average person who tried to do machine learning on AMD GPUs got burned in the past decade and has no reason to change their opinion. Also in the past it was much harder to get access to cutting edge GPUs from AMD. The fact that AMD drops GPU support for ROCm quickly also earns them scorn. I don't think it is an unfair assessment. They earned their reputation.

ROCm has improved a lot. And you can rent mi300x in the cloud now. So if you have a program that runs on Nvidia GPUs, it takes no time to test it on a cloud mi300x. If it works you can use it and save some money in the process.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#255

Post by muxr »

DrNosferatu wrote: post #102
Gasp0de wrote: post #68

Earlier quoted context omitted.

So? If you get twice the RAM at a comparable price and that leads to twice the performance, what's wrong with comparing that?

Nothing wrong - just for transparency. Also, the price difference is not quantified. Additionally, CUDA is a known and tangible software stack - can I try out this "MK1 FLywheel" on my local (AMD) hardware?

None of those things matter, if all you're looking to do is run your existing workloads on mi300x in the cloud. You get more bang per buck by going AMD.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#256

Post by muxr »

DarkmSparks wrote: post #42
triblemaster wrote: post #21

Earlier quoted context omitted.

Blackwell won't be here till next year.

GB200 based on blackwell launched in March of this year. https://www.theregister.com/2024/03/21/nvidia_dgx_gb200_nvk7... MI300X launched 3 months earlier at the end of December. H100 launched March 2023,

You can paper launch as much as you want but Blackwell isn't shipping until Q4.

Also Blackwell's lead will be short lived, because mi350x is coming out next year, and it will have a node and architecture advantage. So AMD will be ahead again.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#258

Post by dexterac »

sigmoid10 wrote: post #214
oelang wrote: post #134

Earlier quoted context omitted.

Microsoft recently announced that they run chatgpt 3.5 & 4 on mi300 on Azure and the price/performance is better. https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd...

I've used ChatGPT on Azure. It sucks on so many levels, everything about it was clearly enforced by some bean counters who see X dollars for Y flops with zero regard for developers. So choosing AMD here would be about par for the course. There is a reason why everyone at the top is racing to buy Nvidia cards and pay the premium.

"Everyone" at top is also developing their own chips for inference and providing APIs for customers to not worry about using CUDA.

It looks like the price to performance of inference tasks gives providers a big incentive to move away from Nvidia.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#259

Post by latchkey »

huevosabio wrote: post #12
rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

RunPod [0] is pricing MI300X at $4.89/hr vs $3.89-4.69/hr for H100s. So, probably around the same price? The tests look promising, though! [0] https://runpod.io/

We are starting at $4.50/hr [0]. The catch is that we won't have availability until mid August.

The weird thing on Runpod is the virtual CPUs, you can't run MI300x in virtual machines yet. It is a missing feature that AMD is working on.

[0] https://hotaisle.xyz/pricing/

Post reply on HN