Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

201–210 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#201

Post by sroussey »

More-nitors wrote: post #165
fleischhauf wrote: post #122

Earlier quoted context omitted.

AMDs deep learning libraries are very bad the last time I checked, nobody uses amd in that space for that reason. Nvidia has a quazi monopoly, that's the main reason for the price difference IMHO.

this... nearly 95% of deeplearning github repos are "tested using cuda gpu - others, not so sure" the only way to out-run nvidia is to have 3~10x better bang-for-buck. Or AMD can just provide a "DIY unlimited gpu RAM upgrade" kit -- a lot of people are buying macstudio 128gb ram because of its "bigger ram-for-buck" than nvidia gpus

I heard apple m4 ultra using 256gb HBM for studio and pro, but I don’t buy it. The 256GB maybe. But a HBM memory control that would go unused on laptops doesn’t pass the smell test.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#202

Post by fvv »

fvv wrote: post #199
tgtweak wrote: post #190

The market (and selling price) is reflecting the perceived value of nvidia's solution vs AMDs - comprehensively including tooling, software, TCO and managability. Also curious how many companies are dropping that much money on those kind of accelerators just to run 8x 7B param models in parallel... You're also talking about being able to train a 14B model on a single accelerator. I'd be curious to see how "full-accel…

the market and the selling price also includes sales strategies, penetrating a sector dominated by a strong player with somewhat "smart" sales strategies *1 and with a growing but certainly less mature product ( expecially software ), it requires suitable pricing and allocation strategies 1. https://www.techspot.com/news/102056-nvidia-allegedly-punish...

the price of h100 reflects and reflected the fact that there is a total monopoly in the training sector,

amd is successfully attacking the inference sector, increasing its advantage with mi325 and aiming for training from 2025 with mi350 (and Infinity Fabric interconnect and other types of interconnection that are arriving for the various topologies), which will probably have an advantage over blackwell, and then fall back against rubin and come back ahead against mi400,

at least, this is what it seems, and as long as the rocm continues to improve.

Personally I am happy to see some competition in the sector and especially on open source software

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#203

Post by mistymountains »

I’m a AI Scientist and train a lot of models. Personally I think AMD is undervalued relative to Nvidia. No, chips aren’t as fast as Nvidia’s latest and yes, there are some hoops to get things working. But for most workloads in most industries (ignoring for the moment that AI is likely a poor use of capital), it will be much more cost effective and achieve about the same results.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#204

Post by Certhas »

kortilla wrote: post #162
Certhas wrote: post #151

Earlier quoted context omitted.

EU GDP per capita 2022 is the same as US GDP per capita 2017. Unless you want to say that the US was much poorer in 2017 than it was in 2022 that's a fairly ridiculous statement. Also, the highest productivity places in the EU have much lower hours worked per capita than the US, with Germans on average working 25% less than Americans and the EU as a whole working 13% less than the US. https://data.oecd.org/emp/hours-…

Yes, if you had 2017 salary in the US today you are much poorer. Inflation was no joke

These numbers are PPP, so inflation adjusted.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#205

Post by erulabs »

CapeTheory wrote: post #64
gizmo wrote: post #39

Earlier quoted context omitted.

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

> Google is not just ads. Bad example given how aggressively they terminate products which don't generate the same revenue as ads. > Apple is not. Best example, they have done a fantastic job of being both a tech company and pseudo-fashion company. > Amazon is not. They don't make anything (at least nothing people want to buy) and have ad revenue as an increase slice of their pie. > Tesla is not. Even bigger hype/spe…

Amazon doesn’t make anything anyone wants to buy?

Not to be snarky, but if AWS counts as “nothing” I’d sure like a slice of nothing please.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#206

Post by mistymountains »

sigmoid10 wrote: post #27
rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.

Unless you develop in CUDA, you can easily train code (e.g. PyTorch) written for training on Nvidia hardware on AMD hardware. You can even keep the .cuda() calls.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#207

Post by mistymountains »

yobbo wrote: post #76
instagraham wrote: post #14

Given that a lot of projects are written or optimised for CUDA, would it require an industry shift if AMD were to become a competitive source of GPUs for AI training?

The model code is comparatively tiny compared to pytorch or CUDA itself. Translating models from CUDA/C could be laborious but not a barrier. Making AMD work effortlessly with pytorch et al should make the switch transparent.

These kinds of comments make me think few people have actually tried. My experience has been 1 work day of getting things set up to work the same as before for training and testing (PyTorch).

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#208

Post by sangnoir »

nabla9 wrote: post #110
m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.

> Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price

What were your thoughts on Zen (1) vs Intel's offerings then? AMD offered more back for the buck then too.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#209

Post by sudosysgen »

almostgotcaught wrote: post #187
sudosysgen wrote: post #176

Earlier quoted context omitted.

LLVM IR to machine code is not the part that AMD has traditionally struggled with. What you call "trivial" is. If everyone started emitting IR and didn't rely on NVidia-owned libs then the space would become unrecognizable. The codegen is something AMD has always been decent at, hence them beating NVidia in compute benchmarks for most of the past 20 years.

> LLVM IR to machine code is not the part that AMD has traditionally struggled with. alright fine it's the codegen and the runtime and the driver and the library ecosystem... > If everyone started emitting IR and didn't rely on NVidia-owned libs then the space would become unrecognizable. I have no clue what this means - which libs are you talking about here? the libs that contain the implementations of their runtime…

I am talking about user and user-level libraries, so from PyTorch to cuBLAS. The rest is currently serviceable and at time was even slightly better than NVidia. If people start shipping code that targets, say, LLVM IR (that then gets converted to PTX or whatever), like one would do using SYCL, we only have to rely the bare minimum.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#210

Post by Certhas »

gizmo wrote: post #156
Certhas wrote: post #151

Earlier quoted context omitted.

EU GDP per capita 2022 is the same as US GDP per capita 2017. Unless you want to say that the US was much poorer in 2017 than it was in 2022 that's a fairly ridiculous statement. Also, the highest productivity places in the EU have much lower hours worked per capita than the US, with Germans on average working 25% less than Americans and the EU as a whole working 13% less than the US. https://data.oecd.org/emp/hours-…

World Bank: European Union gdp per capita for 2022 was $37,433, a 3.33% decline from 2021. U.S. gdp per capita for 2022 was $76,330, a 8.7% increase from 2021. It's not even close?

Sources please? Are these nominal dollars?

World bank data in PPP dollars is reported as 64,600 vs 45,900 here:

https://ourworldindata.org/grapher/gdp-per-capita-worldbank?...

Germany is at 53,900 there, but a good chunk of the difference is simply that US works more per capita. GDP per hour worked is 74$ in the US vs 69$ in Germany, 53$ in Canada. Sweden is ahead of the US. And the EU also includes countries like Bulgaria, which at 29$ is barely ahead of Russias 28$.

https://data.oecd.org/lprdty/gdp-per-hour-worked.htm

France is at 65$ per hour worked, but Germany and France also have significantly lower poverty and inequality rates by any measure you chose, with France more equal than Germany.

The US, of course, remains the dominant economy of the world by any measure. There is no question of that. But the exponential nature of economics, and the structural differences between these different economies, means that GDP numbers compared directly are fairly meaningless.

Edit: That last sentence is too strong as stated. GDP obviously matters a big deal in the grand scheme of things, especially as you jump from lower or middle income to high income countries. But it's all logscale. A factor of 2 is a big deal, a factor of 1.2 might not be.

Post reply on HN