Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

71–80 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#71

Post by huntertwo »

sigmoid10 wrote: post #27
rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.

Large corporate customers like Microsoft and Meta do not use CUDA. They all use custom software. AMD doesn’t have enough GPUs to sell them yet, that’s the real bottleneck.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#72

Post by pell »

earthnail wrote: post #43
raverbashing wrote: post #19

Earlier quoted context omitted.

Yes But there's a long list of German companies not on the DAX (though Germany DAX really deserves to be worth less than NVidia)

The DAX is made of the 40 most valuable German companies. That’s how it is defined. So the companies not in it, again by definition, matter less.

> The DAX is made of the 40 most valuable German companies.

Not to be too nitpicky here but these are only the publicly traded companies. You have a number of pretty large German companies that are still entirely private such as Aldi, Schwarz Group, Boehringer or Bosch.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#73

Post by lycopodiopsida »

Rinzler89 wrote: post #52
jononor wrote: post #48

Earlier quoted context omitted.

Why is producing big companies a goal? High standards of living for all seems to a much better goal. And that can be done with small or big companies - so long as economic production is high enough and distributed well enough.

Because that famous EU welfare is funded via taxes. Having well performing companies funds your welfare system. Currently EU welfare systems are under massive strain and huge waiting lists due to ageing population and economy that hasn't kept up to fund it. There's no free lunch here. You need big companies with scale that pay huge wages as those mean a lot more tax revenue. Saying no to that kind money out of some m…

[flagged]

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#74

Post by dukeyukey »

B1FF_PSUVM wrote: post #38
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> Making real physical things just doesn't scale, Nvidia sells chips ...

Nvidia is a fabless chip designer. They design chips and software to go with the chips, but contract out the actual manufacturing.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#75

Post by dukeyukey »

gizmo wrote: post #39
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

> Europe has almost no real tech companies

How do you define "tech"? Europe's domestic markets are jam-packed full of local tech companies.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#76

Post by yobbo »

instagraham wrote: post #14

Given that a lot of projects are written or optimised for CUDA, would it require an industry shift if AMD were to become a competitive source of GPUs for AI training?

The model code is comparatively tiny compared to pytorch or CUDA itself. Translating models from CUDA/C could be laborious but not a barrier.

Making AMD work effortlessly with pytorch et al should make the switch transparent.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#77

Post by huntertwo »

AMD has better seemingly better hardware - but not the production capacity to compete with Nvidia yet. Will be interesting to see margins compress when real competition catches up.

Everybody thinks it’s CUDA that makes Nvidia the dominant player. It’s not - almost 40% of their revenue this year comes from mega corporations that use their own custom stack to interact with GPUs. It’s only a matter of time before competition catches up and gives us cheaper GPUs.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#78

Post by dukeyukey »

pell wrote: post #62
Rinzler89 wrote: post #45

Earlier quoted context omitted.

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem. Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulati…

Language aside, the entire point of the EU is the single market so you don’t have 26 different rule sets. (There are some exceptions such as health care but that is no different in the US.)

Yes, and it's doing a mediocre-at-best job of it. You can remove tariffs and harmonise regulations, but the EU is trying to unite countries with cultural borders older than Christianity.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#79

Post by dukeyukey »

earthnail wrote: post #43
raverbashing wrote: post #19

Earlier quoted context omitted.

Yes But there's a long list of German companies not on the DAX (though Germany DAX really deserves to be worth less than NVidia)

The DAX is made of the 40 most valuable German companies. That’s how it is defined. So the companies not in it, again by definition, matter less.

No it isn't? It's the list of the biggest blue-chip Germany-HQ'd companies trading on the Frankfurt Stock Exchange. If you're a German company with exclusively German employees trading in London, or Paris, or New York, you don't qualify.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#80

Post by kuschku »

B1FF_PSUVM wrote: post #38
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> Making real physical things just doesn't scale, Nvidia sells chips ...

How much of Nvidia's value do you think comes from the chips? Chips are a commodity, AMD's chips have been better and cheaper than Nvidia for many years.

The reason Nvidia's value has been so inflated is the software stack and the lock-in they offer. CUDA, CuDNN, that's where Nvidia's value lies.

And obviously, now that all relevant ML frameworks are designed for Nvidia's software stack, Nvidia has a monopoly on the supply. That's why their value is being inflated so much.

And Nvidia doesn't have produce the chips themselves, that's all contracted out as well.

Post reply on HN