Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

Post reply on HN
Print view View original

121–130 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#121

Post by almostgotcaught »

pastaguy1 wrote: post #99
huntertwo wrote: post #77

AMD has better seemingly better hardware - but not the production capacity to compete with Nvidia yet. Will be interesting to see margins compress when real competition catches up. Everybody thinks it’s CUDA that makes Nvidia the dominant player. It’s not - almost 40% of their revenue this year comes from mega corporations that use their own custom stack to interact with GPUs. It’s only a matter of time before compet…

Can you explain the cuda-less stack a little more or provide a source?

some people emit llvm ir (maaaaybe ptx) directly instead of using the C/C++ frontend to CUDA. that's absolutely the only optional part of the stack and also basically the most trivial (i.e., it's not the frontend that's hard but the target codegen).

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#122

Post by fleischhauf »

nabla9 wrote: post #110
m_a_g wrote: post #4

"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.

The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.

AMDs deep learning libraries are very bad the last time I checked, nobody uses amd in that space for that reason. Nvidia has a quazi monopoly, that's the main reason for the price difference IMHO.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#123

Post by lycopodiopsida »

Rinzler89 wrote: post #85
lycopodiopsida wrote: post #73

Earlier quoted context omitted.

[flagged]

[flagged]

Stop trolling and throwing around wild accusations. Here are your words:

>You need big companies with scale that pay huge wages as those mean a lot more tax revenue.

Now provide proofs that we need some big corps dodging taxes, including FAANG, and not more small and middle-sized business.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#124

Post by jononor »

gizmo wrote: post #59
jononor wrote: post #48

Earlier quoted context omitted.

Why is producing big companies a goal? High standards of living for all seems to a much better goal. And that can be done with small or big companies - so long as economic production is high enough and distributed well enough.

Because tech innovation requires tons of R&D and you can't afford to do that otherwise. Europeans use American laptops running an American operating system to watch American movies in an American browser. European economic production is nowhere near high enough and now Europe is struggling to provide for its aging population and doesn't have enough good jobs for younger people. I support redistribution generally, but…

To be fair more than half of that laptop hardware is made in China+Taiwan, including a lot of the IP that goes into it. If you look at phones and tablets, there is a bunch of components/IP from European companies also, such as ARM, Bosch, STmicroelectronics, Infineon, NXP etc. Intel has famously struggled and failed multiple times to get into that market. European semiconductor companies are also strong in automotive and other industries that use modern embedded systems. In another consumer electronics niche, a majority of wireless mice and keyboards are build on Nordic Semiconductor chips - from tiny Norway.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#125

Post by pquki4 »

mistermann wrote: post #89
Rinzler89 wrote: post #37

Earlier quoted context omitted.

Same thing was said about Nvidia's crypto bubbles, and then look what happened. Jensen isn't stupid. He's making accelerators for anything so that they'll be ready to catch the next bubble that depends on crazy compute power that can't be done efficiently on CPUs. They're so far the only semi company beating Moore's law by a large margin due to their clever scaling tech while everyone else is like "hey look our new p…

I think there's also a very high prospect of virtual worlds with virtual people (SFW or otherwise) becoming popular, rendered with Apple/META goggles...that could require insane amounts of compute. And this is just one possibility. Relatively cheap multimodal smart glasses you wear when out and around that offload compute to the cloud are another. Nvidia could just as easily triple in short order as get cut in half f…

I thought Meta Horizon and the sales number of Vision Pro[0][1] already proves your thesis wrong. Even Zuckerberg stopped talking about it.

[0]https://www.macrumors.com/2024/04/23/apple-cuts-vision-pro-s... [1] https://www.macrumors.com/2024/04/22/apple-vision-pro-custom...

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#126

Post by amelius »

qeternity wrote: post #93
amelius wrote: post #91

Are these fabbed at the same process node? (Otherwise it's apples and oranges)

It's not apples and oranges. These are the top of the line offerings from the respective companies today.

Perhaps, but that was not the question. After all, these chips are not made by one company. There's a fab too. Not exactly unimportant.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#127

Post by dukeyukey »

Rinzler89 wrote: post #97
leonardovida wrote: post #87

Earlier quoted context omitted.

> There is one exception, founded in 1984 Europe clearly has many problems stimulating investments and creating a competitive environment for startups and tech companies. But you can't say ASML is the only one "real" tech company. What about Adyen, Spotify, Klarna, N26, Revolut, etc?

Most of those companies you mentioned aren't anywhere near as wealthy or as high market caps as US big-tech. Most of them are just payment middlemen not some innovative product nobody else can do, and Spotify survives on monopolizing and squeezing artists, not some innovative product. Kind of like Netflix except Netflix has some cutting edge streaming tech as a product not just IP licenses. ASML is the only product i…

We aren't exclusively talking about big tech here, just tech. Companies like Monzo dominate the local markets because they executed on ideas no-one else tried before. We forgot that huge, international tech companies are pretty much an exclusive American phenomenon, they don't really exist anywhere else.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#128

Post by pell »

dukeyukey wrote: post #118
pell wrote: post #115

Earlier quoted context omitted.

I would argue that the EU is quite a successful organization given the task of setting up general market rules for 26 countries. Obviously there is a lot to criticize about the EU and I can offer you a gigantic list there too. However, I do not see any clear failure of the EU’s approach as a single market so far. Additionally part of the philosophy was establishing peace in a region that was torn up by wars for a lot…

It's definitely successful, and I was probably too harsh there. But I genuinely think the barriers that are left are damn near insurmountable. An awful lot has to change before a Greek tech workers can move to Sweden as easily as a Virginian can move to California.

What would you change if you could? The EU already offers freedom of movement to EU citizens. Of course the US being a country instead of a connection of countries offers a more streamlined experience and surely the shared language plays a huge role too. But when I want to come up with examples such as a Greek person having completely different retirement, health care and legal schemes in Sweden compared to Greece, it seems the US is not so dissimilar there either given that states sometimes have very different approaches to health care, taxes, labor laws, etc.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#129

Post by dukeyukey »

qeternity wrote: post #105
Rinzler89 wrote: post #45

Earlier quoted context omitted.

>How can a wealthy continent with 750 million people produce no big tech companies? It's a big problem. Much more difficult to scale a product across 26 different countries and nearly as many languages and regulatory jurisdictions. US is one country, not a collection of countries fighting each other, meaning your product is instantly available to 300M people speaking the same language under (nearly) the same regulati…

>US is one country, not a collection of countries fighting each other The US is a republic of 50 states. Each state has a huge amount of sovereignty and autonomy. There are 50 state-level regulatory jurisdictions. Not to mention the local-level of government. But in spite of this, the US does not over-regulate. This is the big difference to Europe (I say this as an American expat living in Europe).

American states have generally less autonomy than Canadian provinces do. Federalism is common worldwide, it's far from an American thing. EU countries are _vastly_ more independent than American states, and when you throw in the cultural differences, it grows hugely again.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#130

Post by rfoo »

ta12653421 wrote: post #33
qeternity wrote: post #2

Why the hell are we doing 128 input token benchmarks in 2024. This is not representative of most workloads, and prefill perf is incredibly important.

For understanding: What would be a suitable input length in your oppinion? And why isnt this a good one: Are real-life queries shorter? Or longer? If i count one word as a token, then in my case most of the queries are less than 128 words.

IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.
Post reply on HN