Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

11–20 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#11

Post by epolanski »

m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

Well, there's the beauty of specifying exactly how you ran your benchmark, it is easy to reproduce and disprove or confirm (assuming you got the hardware).

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#12

Post by huevosabio »

rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

RunPod [0] is pricing MI300X at $4.89/hr vs $3.89-4.69/hr for H100s.

So, probably around the same price?

The tests look promising, though!

[0] https://runpod.io/

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#15

Post by impulser_ »

m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

If they used Nvidia's chip would this somehow make the blog post better?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#16

Post by Rinzler89 »

sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#17

Post by irusensei »

instagraham wrote: post #14

Given that a lot of projects are written or optimised for CUDA, would it require an industry shift if AMD were to become a competitive source of GPUs for AI training?

Every hardware vendor is working to provide something with their own technology. I don't know if it's possible but a lot of very resourceful companies are doing their best to break the CUDA dominance. I really hope it works and hopefully a non proprietary standard emerges.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#18

Post by aurareturn »

impulser_ wrote: post #15
m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

If they used Nvidia's chip would this somehow make the blog post better?

For one, they didn't use TensorRT in the test.

Also, stuff like this is hard to take the results seriously:

  * To make an accurate comparison between the systems with different settings of tensor parallelism, we extrapolate throughput for the MI300X by 2.

  * All inference frameworks are configured to use FP16 compute paths. Enabling FP8 compute is left for future work.
They did everything they can to make sure AMD is faster.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#19

Post by raverbashing »

Rinzler89 wrote: post #16
sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Yes

But there's a long list of German companies not on the DAX

(though Germany DAX really deserves to be worth less than NVidia)

Post reply on HN