Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

31–40 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#31

Post by dailykoder »

Rinzler89 wrote: post #16
sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Okay, so you are saying I should move to america, where apparently a lot of people struggle hard to even get a job?

Nah, then ill get my very good wagie pennies here and have plenty jobs available, plus good health insurrance and whatnot.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#33

Post by ta12653421 »

qeternity wrote: post #2

Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.

For understanding:

What would be a suitable input length in your oppinion?

And why isnt this a good one: Are real-life queries shorter? Or longer?

If i count one word as a token, then in my case most of the queries are less than 128 words.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#34

Post by braiamp »

aurareturn wrote: post #18
impulser_ wrote: post #15

Earlier quoted context omitted.

If they used Nvidia's chip would this somehow make the blog post better?

For one, they didn't use TensorRT in the test. Also, stuff like this is hard to take the results seriously: * To make an accurate comparison between the systems with different settings of tensor parallelism, we extrapolate throughput for the MI300X by 2. * All inference frameworks are configured to use FP16 compute paths. Enabling FP8 compute is left for future work. They did everything they can to make sure AMD is f…

I see it as they did everything they can to compare the specific code path. If your workload scales with FP16 but not with tensor cores, then this is the correct way to test. What do you need for LLM inference?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#35

Post by nottorp »

acchow wrote: post #24
nottorp wrote: post #9

Earlier quoted context omitted.

Pretty sure a useful benchmark for this kind of thing would calculate performance per watt (or per watt and dollar). That info is conspicuously absent from the article.

The electricity consumption in the cloud is not really important. The H100 rents for about $4.5/hr consuming 0.7kWh in that hour which will likely cost them less than 7 cents.

> The electricity consumption in the cloud is not really important.

That just says you don't run a cloud for profit :)

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#36

Post by braiamp »

Rinzler89 wrote: post #16
sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Wages is a proxy of how valuable your work is, but not a measure of how value your work is. To support a high salary something has to happen, either the product sold is very expensive or it's being subsidized by investors. No company can pay its employees above what they are able to generate selling the product they worked on indefinitely.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#37

Post by Rinzler89 »

threeseed wrote: post #29
sva_ wrote: post #13

I try to be optimistic about this. Competition is absolutely needed in this space - $NVDA market cap is insane right now, about $0.6 trillion more than the entire Frankfurt Stock Exchange.

We are in the middle of an LLM bubble. Nvidia problem will sort itself out naturally in the coming months/years.

Same thing was said about Nvidia's crypto bubbles, and then look what happened.

Jensen isn't stupid. He's making accelerators for anything so that they'll be ready to catch the next bubble that depends on crazy compute power that can't be done efficiently on CPUs. They're so far the only semi company beating Moore's law by a large margin due to their clever scaling tech while everyone else is like "hey look our new product is 15% more efficient and 15% more IPC than the one we launched 3 years ago".

They may be overvalued now but they definitely won't crash back to their "just gaming GPUs" days.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#38

Post by B1FF_PSUVM »

CapeTheory wrote: post #22
Rinzler89 wrote: post #16

Earlier quoted context omitted.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> Making real physical things just doesn't scale,

Nvidia sells chips ...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#39

Post by gizmo »

CapeTheory wrote: post #22
Rinzler89 wrote: post #16

Earlier quoted context omitted.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms

This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not.

NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments!

Europe has almost no real tech companies. There is one exception, founded in 1984. Not exactly a spring chicken. How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#40

Post by dash2 »

dailykoder wrote: post #31
Rinzler89 wrote: post #16

Earlier quoted context omitted.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Okay, so you are saying I should move to america, where apparently a lot of people struggle hard to even get a job? Nah, then ill get my very good wagie pennies here and have plenty jobs available, plus good health insurrance and whatnot.

German unemployment is 3.2%. US unemployment is 4.0%. Neither of these are at all high by historical standards.

https://www.bls.gov/news.release/empsit.nr0.htm https://www.destatis.de/EN/Press/2024/06/PE24_217_132.html

Post reply on HN