AMD's MI300X Outperforms Nvidia's H100 for LLM Inference • Hacker News Board

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#151

Post by Certhas » Thu, Jun 13, 2024, 1:12 PM UTC

gizmo wrote: post #142

bigfudge wrote: post #132

Earlier quoted context omitted.

And yet standards of living in Europe are comparable to those in the US, and preferable at the median. Our attention is captured by speculative valuations of unicorns, and yet people actually need real stuff made, drugs developed and made etc. Europe does perfectly well in many non winner takes all sectors where English language and network effects are less relevant. The political instability created by the US neolib…

Europe is much much poorer than the US, actually. https://pbs.twimg.com/media/F3PGpsrWEAEiplB?format=jpg&name=...

EU GDP per capita 2022 is the same as US GDP per capita 2017.

Unless you want to say that the US was much poorer in 2017 than it was in 2022 that's a fairly ridiculous statement.

Also, the highest productivity places in the EU have much lower hours worked per capita than the US, with Germans on average working 25% less than Americans and the EU as a whole working 13% less than the US.

https://data.oecd.org/emp/hours-worked.htm

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#152

Post by mark_l_watson » Thu, Jun 13, 2024, 1:13 PM UTC

A good start for AMD. I am also enthusiastic about another non-NVidea inference option: Groq (which I sometimes use).

NVidia relies on TMSC for manufacturing. Samsung is building competing manufacturing infrastructure which is also a good thing, so Taiwan is not a single point of failure.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#153

Post by infamia » Thu, Jun 13, 2024, 1:15 PM UTC

nabla9 wrote: post #131

nabla9 wrote: post #110

Earlier quoted context omitted.

The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.

MTr ------------------ H100 SXM5 80,000 MI300X 153,000 H100 NVL 160,000 H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago. AMD has work to do.

On the other hand, 33% better performance for a 7% increase in power consumption is an appealing bullet point. Lots for AMD to do though, as you said.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#154

Post by gizmo » Thu, Jun 13, 2024, 1:17 PM UTC

lycopodiopsida wrote: post #73

Rinzler89 wrote: post #52

Earlier quoted context omitted.

Because that famous EU welfare is funded via taxes. Having well performing companies funds your welfare system. Currently EU welfare systems are under massive strain and huge waiting lists due to ageing population and economy that hasn't kept up to fund it. There's no free lunch here. You need big companies with scale that pay huge wages as those mean a lot more tax revenue. Saying no to that kind money out of some m…

[flagged]

Yes, this is why European politicians are bending over backwards to get American tech CEOs to open offices in their countries.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#155

Post by dukeyukey » Thu, Jun 13, 2024, 1:18 PM UTC

qeternity wrote: post #100

leonardovida wrote: post #87

Earlier quoted context omitted.

> There is one exception, founded in 1984 Europe clearly has many problems stimulating investments and creating a competitive environment for startups and tech companies. But you can't say ASML is the only one "real" tech company. What about Adyen, Spotify, Klarna, N26, Revolut, etc?

N26, Klarna and Revolut are not tech companies. Neobanks are still banks. And Klarna is just modern store credit cards. These have been around for decades. Adyen is very underrated, and Spotify is definitely tech. Stripe should be on the list. DeepMind at one point.

Revolut is as much a tech company as Netflix or Stripe are - using modern software to fix an old problem.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#156

Post by gizmo » Thu, Jun 13, 2024, 1:23 PM UTC

Certhas wrote: post #151

gizmo wrote: post #142

Earlier quoted context omitted.

Europe is much much poorer than the US, actually. https://pbs.twimg.com/media/F3PGpsrWEAEiplB?format=jpg&name=...

EU GDP per capita 2022 is the same as US GDP per capita 2017. Unless you want to say that the US was much poorer in 2017 than it was in 2022 that's a fairly ridiculous statement. Also, the highest productivity places in the EU have much lower hours worked per capita than the US, with Germans on average working 25% less than Americans and the EU as a whole working 13% less than the US. https://data.oecd.org/emp/hours-…

World Bank:

European Union gdp per capita for 2022 was $37,433, a 3.33% decline from 2021.

U.S. gdp per capita for 2022 was $76,330, a 8.7% increase from 2021.

It's not even close?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#157

Post by nextworddev » Thu, Jun 13, 2024, 1:23 PM UTC

We need more competition in the training space, not inference.

For consumer grade inference, there's already many options available.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#158

Post by sumedh » Thu, Jun 13, 2024, 1:26 PM UTC

viraptor wrote: post #66

Rinzler89 wrote: post #50

Earlier quoted context omitted.

Maybe but it's not like all those AI compute units or whatever Nvidia called them will be thrown in the dumpster after the AI bubble pops. There's a lot of problems the can be solved on them and researcher are always looking for new problems to solve as compute becomes accesibile. I'm tired of hearing about Nvidia's "luck". There was no luck involved. Nvidia shiped Cuda on consumer GPUs since 2006. That's almost 20 y…

They will not be thrown in the dumpster, but that's actually a bad thing for NVIDIA. We had a very short period when lots of miners dumped their RTX cards on ebay and the prices fell a lot for some time. (then AI on RTX became a thing at small scales) When the A100/H100s get replaced, they will flood the market. There's many millions of $ stuck in those assets right now and in a few years they will dominate research…

> they're screwed and the compute market is saturated for years

If all this extra computing power is available, smart people will find a way to use it somehow.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#159

Post by fisf » Thu, Jun 13, 2024, 1:26 PM UTC

nabla9 wrote: post #149

lhl wrote: post #141

Earlier quoted context omitted.

Isn't SXM5 higher bandwidth? It's 900 GB/s of bidirectional bandwidth per GPU across 18 NVLink 4 channels. The NVL's are on PCIe 5, and even w/ NVLink only get to 600 GB/s of bandwidth across 3 NVLink bridges (across only pairs of cards)? I haven't done a head to head and I suppose it depends on whether tensor parallelism actually scales linearly or not, but my understanding is since the NVL's are just PCIe/NVLink pa…

> but I think it's one that's mostly about Nvidia's platform dominance and profit margins more Profit margins and dominance are result from performance, not the other way around. It does not matter if Nvidia tools are better when you deploy large number of chips for inference and it does more flops per watt or second. It's seller market and if AMD can't ask high price, their chip do not perform. ---- Question: People…

> People here seem to think that Nvidia has absolutely no advantage in their microarchitecture design skills. It's all in software or monopoly.

That's an extrapolation. Microarchitecture design skills are not theoretical numbers you manage to put on a spec sheet. You cannot decouple the software driving the hardware - that's not a trivial problem.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#160

Post by ericd » Thu, Jun 13, 2024, 1:27 PM UTC

lycopodiopsida wrote: post #73

Rinzler89 wrote: post #52

Earlier quoted context omitted.

Because that famous EU welfare is funded via taxes. Having well performing companies funds your welfare system. Currently EU welfare systems are under massive strain and huge waiting lists due to ageing population and economy that hasn't kept up to fund it. There's no free lunch here. You need big companies with scale that pay huge wages as those mean a lot more tax revenue. Saying no to that kind money out of some m…

[flagged]

Even if a company doesn’t pay taxes, its masses of highly paid workers do. Each of those SWEs making $300k+ are paying more in taxes than the entire earnings of the average EU dev.