AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
blog.tensorwave.com
AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
1–10 of 273 posts
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#2Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#3Is this an ad for a new, closed-source, GPGPU backend?
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#4"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications."
I suggest taking the report with a grain of salt.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#5> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM.
> MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16
> Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM.
> H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16
I really wonder about the pricing. In theory the MI300X is supposed to be cheaper, but whether is that is really the case in practice remains to be seen.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#6Is this an ad for a new, closed-source, GPGPU backend?
Pretty much and the test suit is optimized to get the results they wanted.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#7I'm wondering if the tensor parallel settings have any impact on the performance. My naive guess is yes but not sure.
According to the article: """ AMD Configuration: Tensor parallelism set to 1 (tp=1), since we can fit the entire model Mixtral 8x7B in a single MI300X’s 192GB of VRAM.
NVIDIA Configuration: Tensor parallelism set to 2 (tp=2), which is required to fit Mixtral 8x7B in two H100’s 80GB VRAM. """
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#8hopper (H100) is the predecessor to the current blackwell architecture.
This is a new AMD vs last generation nvidia benchmark.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#9Is this an ad for a new, closed-source, GPGPU backend?
Pretty much and the test suit is optimized to get the results they wanted.
Pretty sure a useful benchmark for this kind of thing would calculate performance per watt (or per watt and dollar).
That info is conspicuously absent from the article.
Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference
#10INT8/FP8 benchmarks would've been great, both cards could have loaded them with around 60GB VRAM instead of TP=2 on H100.