Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

41–50 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#41

Post by chad1n »

B1FF_PSUVM wrote: post #38
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> Making real physical things just doesn't scale, Nvidia sells chips ...

Investors don't even know what NVIDIA is selling, I was listening to a random investor podcast and they were talking about Intel, AMD and NVIDIA, but no one knew what exactly they are selling, they only knew they are part of this AI bubble so that's why you should invest in them

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#42

Post by DarkmSparks »

triblemaster wrote: post #21
DarkmSparks wrote: post #8

hopper (H100) is the predecessor to the current blackwell architecture. This is a new AMD vs last generation nvidia benchmark.

Blackwell won't be here till next year.

GB200 based on blackwell launched in March of this year.

https://www.theregister.com/2024/03/21/nvidia_dgx_gb200_nvk7...

MI300X launched 3 months earlier at the end of December.

H100 launched March 2023,

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#43

Post by earthnail »

raverbashing wrote: post #19
Rinzler89 wrote: post #16

Earlier quoted context omitted.

It's more how little the Frankfurt stock Exchange is worth. And European devs keep wondering why our wages are lower than in the US for the same work. That's why.

Yes But there's a long list of German companies not on the DAX (though Germany DAX really deserves to be worth less than NVidia)

The DAX is made of the 40 most valuable German companies. That’s how it is defined. So the companies not in it, again by definition, matter less.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#45

Post by Rinzler89 »

gizmo wrote: post #39
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem.

Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulations.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#46

Post by spacecadet »

ta12653421 wrote: post #33
qeternity wrote: post #2

Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.

For understanding: What would be a suitable input length in your oppinion? And why isnt this a good one: Are real-life queries shorter? Or longer? If i count one word as a token, then in my case most of the queries are less than 128 words.

In most cases thats not enough

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#47

Post by viraptor »

Rinzler89 wrote: post #37
threeseed wrote: post #29

Earlier quoted context omitted.

We are in the middle of an LLM bubble. Nvidia problem will sort itself out naturally in the coming months/years.

Same thing was said about Nvidia's crypto bubbles, and then look what happened. Jensen isn't stupid. He's making accelerators for anything so that they'll be ready to catch the next bubble that depends on crazy compute power that can't be done efficiently on CPUs. They're so far the only semi company beating Moore's law by a large margin due to their clever scaling tech while everyone else is like "hey look our new p…

They got extremely lucky with AI following crypto. The timing was close to perfect. I'm not sure there will be another wave like that at all for a long while.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#48

Post by jononor »

gizmo wrote: post #39
CapeTheory wrote: post #22

Earlier quoted context omitted.

The DAX is only 40 companies, most of which make real products rather than advertising mechanisms. Making real physical things just doesn't scale, and never will. While I would enjoy a US tech salary, I'm not sure we want a world where all manufacturing is set aside to focus on the attention economy. Nvidia value deserves to be much higher than any company on the DAX (maybe all of them together, as it currently is) -…

> The DAX is only 40 companies, most of which make real products rather than advertising mechanisms This, as the kids say, is just cope. American big tech makes real products. Google is not just ads. Apple is not. Amazon is not. Tesla is not. NVidia is not. Netflix is not. NVidia might be overvalued because of the current AI hype but that does not diminish their real accomplishments! Europe has almost no real tech co…

Why is producing big companies a goal? High standards of living for all seems to a much better goal. And that can be done with small or big companies - so long as economic production is high enough and distributed well enough.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#49

Post by hyperman1 »

sigmoid10 wrote: post #27
rjzzleep wrote: post #5

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the…

It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.

And this shouldn't be to hard if you know the ins and outs of the hardware and have a reasonable dev team. So why aren't they doing it?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#50

Post by Rinzler89 »

viraptor wrote: post #47
Rinzler89 wrote: post #37

Earlier quoted context omitted.

Same thing was said about Nvidia's crypto bubbles, and then look what happened. Jensen isn't stupid. He's making accelerators for anything so that they'll be ready to catch the next bubble that depends on crazy compute power that can't be done efficiently on CPUs. They're so far the only semi company beating Moore's law by a large margin due to their clever scaling tech while everyone else is like "hey look our new p…

They got extremely lucky with AI following crypto. The timing was close to perfect. I'm not sure there will be another wave like that at all for a long while.

Maybe but it's not like all those AI compute units or whatever Nvidia called them will be thrown in the dumpster after the AI bubble pops. There's a lot of problems the can be solved on them and researcher are always looking for new problems to solve as compute becomes accesibile.

I'm tired of hearing about Nvidia's "luck". There was no luck involved. Nvidia shiped Cuda on consumer GPUs since 2006. That's almost 20 years time researchers had to find used cases for that compute and Nvidia made it possible. In other words the AI bubble happened because Nvidia made the necessary ground work for it to happen, they didn't just fall into it by luck.

Post reply on HN