Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

61–70 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#61

Post by renonce »

michaelnny wrote: post #7

I'm wondering if the tensor parallel settings have any impact on the performance. My naive guess is yes but not sure. According to the article: """ AMD Configuration: Tensor parallelism set to 1 (tp=1), since we can fit the entire model Mixtral 8x7B in a single MI300X’s 192GB of VRAM. NVIDIA Configuration: Tensor parallelism set to 2 (tp=2), which is required to fit Mixtral 8x7B in two H100’s 80GB VRAM. """

I personally find such comparisons unfair. A good comparison should optimize for each device configuration, which means use a model within the VRAM limit and quantize to 8 bits where it boosts performance etc and avoid shortcomings of both devices unless necessary.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#62

Post by pell »

Rinzler89 wrote: post #45
gizmo wrote: post #39

Earlier quoted context omitted.

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem. Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulati…

Language aside, the entire point of the EU is the single market so you don’t have 26 different rule sets. (There are some exceptions such as health care but that is no different in the US.)

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#64

Post by CapeTheory »

gizmo wrote: post #39
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

> Google is not just ads.

Bad example given how aggressively they terminate products which don't generate the same revenue as ads.

> Apple is not.

Best example, they have done a fantastic job of being both a tech company and pseudo-fashion company.

> Amazon is not.

They don't make anything (at least nothing people want to buy) and have ad revenue as an increase slice of their pie.

> Tesla is not.

Even bigger hype/speculation vehicle than Nvidia.

> NVidia is not.

Nvidia of 5 years ago would not have appeared on this list, being too much of a niche tech company. Good at what they do, but hugely hype-fuelled.

> Netflix is not.

Running out of growth potential with their current business model, starting to introduce ads!

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#65

Post by yobbo »

Rinzler89 wrote: post #37
threeseed wrote: post #29

Earlier quoted context omitted.

We are in the middle of an LLM bubble. Nvidia problem will sort itself out naturally in the coming months/years.

Same thing was said about Nvidia's crypto bubbles, and then look what happened. Jensen isn't stupid. He's making accelerators for anything so that they'll be ready to catch the next bubble that depends on crazy compute power that can't be done efficiently on CPUs. They're so far the only semi company beating Moore's law by a large margin due to their clever scaling tech while everyone else is like "hey look our new p…

Intel's all time high was in 2000, if I'm reading the charts correctly.

Of course the equivalent can happen to Nvidia. Seems almost certain.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#66

Post by viraptor »

Rinzler89 wrote: post #50
viraptor wrote: post #47

Earlier quoted context omitted.

They got extremely lucky with AI following crypto. The timing was close to perfect. I'm not sure there will be another wave like that at all for a long while.

Maybe but it's not like all those AI compute units or whatever Nvidia called them will be thrown in the dumpster after the AI bubble pops. There's a lot of problems the can be solved on them and researcher are always looking for new problems to solve as compute becomes accesibile. I'm tired of hearing about Nvidia's "luck". There was no luck involved. Nvidia shiped Cuda on consumer GPUs since 2006. That's almost 20 y…

They will not be thrown in the dumpster, but that's actually a bad thing for NVIDIA. We had a very short period when lots of miners dumped their RTX cards on ebay and the prices fell a lot for some time. (then AI on RTX became a thing at small scales) When the A100/H100s get replaced, they will flood the market. There's many millions of $ stuck in those assets right now and in a few years they will dominate research and top end of hobbies. Only high profile companies/universities/researchers will look at buying anything newer. Maybe NVIDIA can do 2 generations of those cards, but ASIC-based solutions will hit the market and the generic CUDA will become a problem rather than a blessing. Same story as BTC miners and graphics cards.

Sure, they didn't get lucky with the tech they had to offer - that was well developed for years. They just got lucky that the next big thing was compute-based. If the next thing is memory/storage-based, they're screwed and the compute market is saturated for years - they have only gamers left.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#67

Post by stefs »

ta12653421 wrote: post #33
qeternity wrote: post #2

Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.

For understanding: What would be a suitable input length in your oppinion? And why isnt this a good one: Are real-life queries shorter? Or longer? If i count one word as a token, then in my case most of the queries are less than 128 words.

It's not just the current prompt, but the whole conversation, if possible. Or, if you want the AI to summarise an article, the article has to fit in.

If I understood that correctly, context length is something like session storage or short term memory. If it's too small the AI starts to forget what it's talking about.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#68

Post by Gasp0de »

DrNosferatu wrote: post #51

The comparison is between setups with different amounts of GPU RAM and there's no quantification of final performance/price.

So? If you get twice the RAM at a comparable price and that leads to twice the performance, what's wrong with comparing that?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#69

Post by Rinzler89 »

pell wrote: post #62
Rinzler89 wrote: post #45

Earlier quoted context omitted.

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem. Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulati…

Language aside, the entire point of the EU is the single market so you don’t have 26 different rule sets. (There are some exceptions such as health care but that is no different in the US.)

Except you do have 26 different rules.

It's a single market on paper as the eu only mandates a small subset of common rules and regulations such as removing tarrifs or freedom of movement, but have you ever tried in practice to launch your company from Belgium to France or from Netherlands to Belgium or from Austria to Germany, or from Romania to Italy?

It's much more difficult when the rubber hits the road as every country has various extra laws and protectionist measures in place to protect it's domestic players from outsiders even if they came from within the EU. And that's besides the language barrier which means added costs. This is much less efficient than the US market.

EU countries and voters still value their national sovereignty and culture (both with the upsides and downsides) above a united EU under the same laws and language for everyone, ruled from outside their country's borders. See what happened with Brexit and the constant internal squabbling and sabotaging over critical EU issues that affect us all like the war in Ukraine or illegal mass migration. An US style unification just won't work here since every little country wants to be it's own king while having its cake and eating it too.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#70

Post by CapeTheory »

B1FF_PSUVM wrote: post #38
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> Making real physical things just doesn't scale, Nvidia sells chips ...

They can't make them fast enough to satisfy hype-driven demand; they are not scaling like a tech company, but their market cap is being inflated like one.
Post reply on HN