Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

101–110 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#101

Post by gizmo »

dukeyukey wrote: post #75
gizmo wrote: post #39

Earlier quoted context omitted.

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

> Europe has almost no real tech companies How do you define "tech"? Europe's domestic markets are jam-packed full of local tech companies.

The grandparent claimed that the DAX 40 has real businesses that make things as opposed to American tech businesses that just serve the attention economy. Clearly not true. European businesses run on American technology. American businesses do not run on European technology.

Europe has many small and not very profitable tech companies. Almost no large and profitable ones. https://pbs.twimg.com/media/GNDtCtTXcAAiwFk?format=jpg&name=...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#102

Post by DrNosferatu »

Gasp0de wrote: post #68
DrNosferatu wrote: post #51

The comparison is between setups with different amounts of GPU RAM and there's no quantification of final performance/price.

So? If you get twice the RAM at a comparable price and that leads to twice the performance, what's wrong with comparing that?

Nothing wrong - just for transparency.

Also, the price difference is not quantified.

Additionally, CUDA is a known and tangible software stack - can I try out this "MK1 FLywheel" on my local (AMD) hardware?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#103

Post by Certhas »

CapeTheory wrote: post #64
gizmo wrote: post #39

Earlier quoted context omitted.

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

> Google is not just ads. Bad example given how aggressively they terminate products which don't generate the same revenue as ads. > Apple is not. Best example, they have done a fantastic job of being both a tech company and pseudo-fashion company. > Amazon is not. They don't make anything (at least nothing people want to buy) and have ad revenue as an increase slice of their pie. > Tesla is not. Even bigger hype/spe…

Nvidia's valuation is in part driven by the fact that their chips potentially disrupt the vehicle for ad revenue that is search.

For all his insanity, the one thing I respect Musk for, is that he actually started successful companies that make stuff. Creating a new car manufacturer of the scale of BMW out of nothing was widely considered impossible before.

Of course he did this from a position of extreme wealth, but none of his peers managed to do that. Everyone else is just seeking rent by trying to be first to implement some tech transition that is coming anyway. And that might be a lot more valuable to society if it was managed differently...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#104

Post by idiliv »

scotty79 wrote: post #96
epolanski wrote: post #11

Earlier quoted context omitted.

Well, there's the beauty of specifying exactly how you ran your benchmark, it is easy to reproduce and disprove or confirm (assuming you got the hardware).

As easy as getting yourself 8 H100 and 8 MI300X. Fun weekend project for anybody.

You can rent them online for ~ 4-5 $ per hour per GPU. Not cheap, but definitely feasible as a weekend project.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#105

Post by qeternity »

Rinzler89 wrote: post #45
gizmo wrote: post #39

Earlier quoted context omitted.

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem. Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulati…

>US is one country, not a collection of countries fighting each other

The US is a republic of 50 states. Each state has a huge amount of sovereignty and autonomy. There are 50 state-level regulatory jurisdictions. Not to mention the local-level of government.

But in spite of this, the US does not over-regulate. This is the big difference to Europe (I say this as an American expat living in Europe).

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#106

Post by HPsquared »

braiamp wrote: post #34
aurareturn wrote: post #18

Earlier quoted context omitted.

For one, they didn't use TensorRT in the test. Also, stuff like this is hard to take the results seriously: * To make an accurate comparison between the systems with different settings of tensor parallelism, we extrapolate throughput for the MI300X by 2. * All inference frameworks are configured to use FP16 compute paths. Enabling FP8 compute is left for future work. They did everything they can to make sure AMD is f…

I see it as they did everything they can to compare the specific code path. If your workload scales with FP16 but not with tensor cores, then this is the correct way to test. What do you need for LLM inference?

Couldn't they find a real workload that does this?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#107

Post by chollida1 »

CapeTheory wrote: post #22
Rinzler89 wrote: post #16

Earlier quoted context omitted.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> but how much of that current value is real rather than an AI speculation bubble?

I mean, by definition given that it trades freely their market cap is real. Your market cap today is what the market thinks your future cash flows are worth. The bubble and the bubble popping should in theory both be priced into Nvidia's market cap.

What isnt' is events the market doesn't anticipate, AMD coming out with a current generation chip that can do inference as well as the H100 is something the market hasn't priced in.

Andy our manufacturing example is very poor as NVidia is certainly part of the manufacturing pipe line by designing physical products that people buy.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#108

Post by zzbn00 »

dukeyukey wrote: post #74
B1FF_PSUVM wrote: post #38

Earlier quoted context omitted.

> Making real physical things just doesn't scale, Nvidia sells chips ...

Nvidia is a fabless chip designer. They design chips and software to go with the chips, but contract out the actual manufacturing.

Yes, they buy these chips from fabs (made to their design) but it is the chips they then sell on, at about a 4x markup to what they bought them for. Good business model! Buy low, sell high. Whether good enough to justify current valuation is another question.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#109

Post by gizmo »

qeternity wrote: post #100
leonardovida wrote: post #87

Earlier quoted context omitted.

> There is one exception, founded in 1984 Europe clearly has many problems stimulating investments and creating a competitive environment for startups and tech companies. But you can't say ASML is the only one "real" tech company. What about Adyen, Spotify, Klarna, N26, Revolut, etc?

N26, Klarna and Revolut are not tech companies. Neobanks are still banks. And Klarna is just modern store credit cards. These have been around for decades. Adyen is very underrated, and Spotify is definitely tech. Stripe should be on the list. DeepMind at one point.

Stripe was founded in California. It's an American business that focused exclusively on the American domestic market in their first years of operation. Many tech companies in the US are founded by immigrants from Europe and elsewhere. That the Collisons chose to start their business in the States is no coincidence.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#110

Post by nabla9 »

m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

The salt is in the plain sight.

The do the standard AMD comparison:

  8x AMD MI300X (192GB, 750W) GPU  
  8x H100 SXM5 (80GB, 700W) GPU
The fair comparison would be against

  8x H100 NVL (188GB,
Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.
Post reply on HN