Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

131–140 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#131

Post by nabla9 »

nabla9 wrote: post #110
m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.

                 MTr
  ------------------
  H100 SXM5   80,000 
  MI300X     153,000
  H100 NVL   160,000
H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago.

AMD has work to do.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#132

Post by bigfudge »

gizmo wrote: post #59
jononor wrote: post #48

Earlier quoted context omitted.

Why is producing big companies a goal? High standards of living for all seems to a much better goal. And that can be done with small or big companies - so long as economic production is high enough and distributed well enough.

Because tech innovation requires tons of R&D and you can't afford to do that otherwise. Europeans use American laptops running an American operating system to watch American movies in an American browser. European economic production is nowhere near high enough and now Europe is struggling to provide for its aging population and doesn't have enough good jobs for younger people. I support redistribution generally, but…

And yet standards of living in Europe are comparable to those in the US, and preferable at the median. Our attention is captured by speculative valuations of unicorns, and yet people actually need real stuff made, drugs developed and made etc. Europe does perfectly well in many non winner takes all sectors where English language and network effects are less relevant. The political instability created by the US neoliberal experiment is something I hope we can avoid over here too.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#133

Post by nabla9 »

fleischhauf wrote: post #122
nabla9 wrote: post #110

Earlier quoted context omitted.

The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.

AMDs deep learning libraries are very bad the last time I checked, nobody uses amd in that space for that reason. Nvidia has a quazi monopoly, that's the main reason for the price difference IMHO.

> that's the main reason for the price difference IMHO.

Explain why the performance difference does not matter?

AMD does only 33% better with a chip that has 2X transistors and 2X memory.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#134

Post by oelang »

sigmoid10 wrote: post #27
rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.

Microsoft recently announced that they run chatgpt 3.5 & 4 on mi300 on Azure and the price/performance is better.

https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#135

Post by Vvector »

qeternity wrote: post #93
amelius wrote: post #91

Are these fabbed at the same process node? (Otherwise it's apples and oranges)

It's not apples and oranges. These are the top of the line offerings from the respective companies today.

Nvidia just started shipping the H200 to selected companies two months ago. Too late for this benchmark.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#136

Post by dukeyukey »

pell wrote: post #128
dukeyukey wrote: post #118

Earlier quoted context omitted.

It's definitely successful, and I was probably too harsh there. But I genuinely think the barriers that are left are damn near insurmountable. An awful lot has to change before a Greek tech workers can move to Sweden as easily as a Virginian can move to California.

What would you change if you could? The EU already offers freedom of movement to EU citizens. Of course the US being a country instead of a connection of countries offers a more streamlined experience and surely the shared language plays a huge role too. But when I want to come up with examples such as a Greek person having completely different retirement, health care and legal schemes in Sweden compared to Greece, i…

Mostly stuff you can't really change. Like, the cultural difference between Sweden and Greece is _way_ bigger than between California and Virginia. You've got the language barrier too. This isn't gonna be fixed, maybe ever, but will continue to cause friction.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#137

Post by mistermann »

pquki4 wrote: post #125
mistermann wrote: post #89

Earlier quoted context omitted.

I think there's also a very high prospect of virtual worlds with virtual people (SFW or otherwise) becoming popular, rendered with Apple/META goggles...that could require insane amounts of compute. And this is just one possibility. Relatively cheap multimodal smart glasses you wear when out and around that offload compute to the cloud are another. Nvidia could just as easily triple in short order as get cut in half f…

I thought Meta Horizon and the sales number of Vision Pro[0][1] already proves your thesis wrong. Even Zuckerberg stopped talking about it. [0] https://www.macrumors.com/2024/04/23/apple-cuts-vision-pro-s... [1] https://www.macrumors.com/2024/04/22/apple-vision-pro-custom...

I am referring to the future (the actual future, not simulated ones), so it is not possible to know if I am wrong.

I predict this is yet another domain rich with opportunity for AI.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#138

Post by dukeyukey »

jononor wrote: post #86
dukeyukey wrote: post #81

Earlier quoted context omitted.

Big companies means efficiencies of scale. Small companies that succeed and grow inevitably become big companies. If they don't, it means the qualities that make them effective don't scale to the rest of the economy.

They become big but not necessarily huge - like the 10 biggest tech companies that people here put as the benchmark. For that one needs organizations that also continuously increases their scope - going into new markets, consolidating exiting markets, buying up existing players. And if they are to continue being "European" then they must resist being bought up by the huge US or global tech companies. The latter is a…

Yep, that's a huge problem, and one that isn't easy to fix. The European market is vastly more gragmented than the North American one, so even without Europe's penchant for taxes and regulation, North America can more easily get giant companies that can reach across the Atlantic.

The only real answer is protectionism, and there's a good chance that'll hurt more than it helps.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#139

Post by littlecranky67 »

sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

Frankfurt Stock Exchange or the DAX is mostly irrelevant. Germany has a strong, family-owned Mittelstand, those companies are not publicly traded and thus not listed. Plus, we have some giants that are also not publicly listed but belong to the richest Germans (Lidl, Aldi of discount groceries, but also automotive OEM Bosch).

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#140

Post by petra »

chad1n wrote: post #41
B1FF_PSUVM wrote: post #38

Earlier quoted context omitted.

> Making real physical things just doesn't scale, Nvidia sells chips ...

Investors don't even know what NVIDIA is selling, I was listening to a random investor podcast and they were talking about Intel, AMD and NVIDIA, but no one knew what exactly they are selling, they only knew they are part of this AI bubble so that's why you should invest in them

What about seroous analysts, say like morningstar, don't the undedtand the business that they recommended relatively well?
Post reply on HN