Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

191–200 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#191

Post by Rinzler89 »

gizmo wrote: post #174
Rinzler89 wrote: post #170

Earlier quoted context omitted.

GDP per capita as a stand alone metric doesn't mean much.

All rich countries have high GDP per capita and all poor countries have low GDP per capita. Zero exceptions. Despite the shortcomings of GDP as a metric it still tracks prosperity very accurately.

You can't get any more obvious than that, but that wasn't my point to say that higher GDP doesn't make you richer than a low GDP, but to say GDP/capita as a number alone is not a measure of wealth, income or prosperity between countries, even in the EU.

For example Ireland has by a long margin the highest GDP/capita in the whole EU, and it would make you think the average Irish worker earns more that any other worker in the EU and drives a Lambo, but that's not what's happening. It's because most US corporations funnel their EU money through their Irish holding companies skewing the statistic.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#192

Post by threecheese »

preisschild wrote: post #185
avidphantasm wrote: post #182

Earlier quoted context omitted.

Maybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.

But Apple doesn't produce servers or server hardware.

Good point; if the Mx architecture does prove to be a viable competitor to Nvidia/AMD for training and/or inference, do you think Apple would enter the server market? They continue to diversify on the consumer side, I wonder if they have their eye on the business market; I am not sure if their general strategy of “prettier is better” would work well there though.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#193

Post by zhyder »

Shouldn't the right benchmark be performance per watt? It's easy enough to add more chips to do LLM training or inference in parallel.

Maybe the benchmark should be performance per $... though I suspect power consumption will eclipse the cost of purchasing the chips from NVDA or AMD (and costs of chips will vary over time and with discounts). EDIT: was wrong on eclipsing; still am looking for a more durable benchmark (performance per billion transistors?) given it's suspected NVDA's chips are over-priced due to demand outstripping supply for now, and AMD's are under- to get a foothold in this market.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#194

Post by [deleted] »

avidphantasm wrote: post #182
nabla9 wrote: post #131

Earlier quoted context omitted.

MTr ------------------ H100 SXM5 80,000 MI300X 153,000 H100 NVL 160,000 H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago. AMD has work to do.

Maybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.

[deleted]

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#195

Post by mmoskal »

zhyder wrote: post #193

Shouldn't the right benchmark be performance per watt? It's easy enough to add more chips to do LLM training or inference in parallel. Maybe the benchmark should be performance per $... though I suspect power consumption will eclipse the cost of purchasing the chips from NVDA or AMD (and costs of chips will vary over time and with discounts). EDIT: was wrong on eclipsing; still am looking for a more durable benchmark…

Not quite. Assume 1kW power consumption (with cooling). At $0.08/kWh (avarage US industrial rate) this is $700 per year. Adjust for more cooling etc and for say 5 years of usage but you still won't be anywhere near the $25k MSRP for H100.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#197

Post by fvv »

tgtweak wrote: post #190

The market (and selling price) is reflecting the perceived value of nvidia's solution vs AMDs - comprehensively including tooling, software, TCO and managability. Also curious how many companies are dropping that much money on those kind of accelerators just to run 8x 7B param models in parallel... You're also talking about being able to train a 14B model on a single accelerator. I'd be curious to see how "full-accel…

mi300x win in some inference workloads, h100 win in training and some others inference workloads ( fp8 inference with tensorRT-llm , rocm is young but is growing fast )

in a single system ( 8x accelerators ) LLMs, mi300x has very competitive inference TCO vs h100 .

also :

AMD Instinct MI300X Offers The Best Price To Performance on GPT-4 According To Microsoft, Red Team On-Track For 100x Perf/Watt By 2027

https://wccftech.com/amd-instinct-mi300x-best-price-performa...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#198

Post by gizmo »

Rinzler89 wrote: post #191
gizmo wrote: post #174

Earlier quoted context omitted.

All rich countries have high GDP per capita and all poor countries have low GDP per capita. Zero exceptions. Despite the shortcomings of GDP as a metric it still tracks prosperity very accurately.

You can't get any more obvious than that, but that wasn't my point to say that higher GDP doesn't make you richer than a low GDP, but to say GDP/capita as a number alone is not a measure of wealth, income or prosperity between countries, even in the EU. For example Ireland has by a long margin the highest GDP/capita in the whole EU, and it would make you think the average Irish worker earns more that any other worker…

> For example Ireland has by a long margin the highest GDP/capita in the whole EU

No, Luxembourg does.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#199

Post by fvv »

tgtweak wrote: post #190

The market (and selling price) is reflecting the perceived value of nvidia's solution vs AMDs - comprehensively including tooling, software, TCO and managability. Also curious how many companies are dropping that much money on those kind of accelerators just to run 8x 7B param models in parallel... You're also talking about being able to train a 14B model on a single accelerator. I'd be curious to see how "full-accel…

the market and the selling price also includes sales strategies, penetrating a sector dominated by a strong player with somewhat "smart" sales strategies *1

and with a growing but certainly less mature product ( expecially software ), it requires suitable pricing and allocation strategies

1. https://www.techspot.com/news/102056-nvidia-allegedly-punish...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#200

Post by sroussey »

preisschild wrote: post #185
avidphantasm wrote: post #182

Earlier quoted context omitted.

Maybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.

But Apple doesn't produce servers or server hardware.

They should buy Grok (the hardware inference company, not the lame twitter bot).
Post reply on HN