Live data from Hacker News

AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

blog.tensorwave.com

231–240 of 273 posts

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#231

Earlier quoted context omitted.

.... while all of the content is hosted on Linux servers, and you're listening to music through a Swedish app. That is running on silicone made in Taiwan. Using equipment that can currently only be manufactured in Belgium and Germany. On a Mac you are using a British instruction set. Also in terms of tech innovation: What part of the US-based tech innovation couldn't have been (and actually were) achieved with open-s…

>EU per Capita GDP in 2022 is the same as USA 2016. Was the USA in 2016 struggling but now isn't? Absolutely the US would be struggling if the GDP was still 2016 values with today’s costs.

[deleted]

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#232

Earlier quoted context omitted.

>EU per Capita GDP in 2022 is the same as USA 2016. Was the USA in 2016 struggling but now isn't? Absolutely the US would be struggling if the GDP was still 2016 values with today’s costs.

Pretty much. I'd also love it (European) if we stil had 2017's rent prices and consumer prices, but we don't. Housing, energy, food, healthcare and everything has gone up like crazy. We're definitely poorer than before, just sweeping it under the rug pretending everyting's fine. So it boggles my mind that your parent tried to make a point by equating USA 2016 with EU 2022 GDP/capita as if nothing's wrong with that. A…

The 2017 number was a mistake by me, I wanted to look at inflation adjusted/PPP, and that was nominal. You can clearly see the massive effect of the Russian invasion of Ukraine and the resulting inflation in the data. If we were looking at nominal values, the opposite is the case: Nominally GDP is rising rapidly:

Nominal: https://data.oecd.org/gdp/gross-domestic-product-gdp.htm

PPP/Inflation adjusted not so much: https://ourworldindata.org/grapher/gdp-per-capita-worldbank?...

But pretty much all countries are still well ahead of where we were in 2017. If you feel poorer than in 2017 it's because you're getting less of a larger pie, not because the economy is producing less than it did then.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#233
post #59

Earlier quoted context omitted.

Because tech innovation requires tons of R&D and you can't afford to do that otherwise. Europeans use American laptops running an American operating system to watch American movies in an American browser. European economic production is nowhere near high enough and now Europe is struggling to provide for its aging population and doesn't have enough good jobs for younger people. I support redistribution generally, but…

.... while all of the content is hosted on Linux servers, and you're listening to music through a Swedish app. That is running on silicone made in Taiwan. Using equipment that can currently only be manufactured in Belgium and Germany. On a Mac you are using a British instruction set. Also in terms of tech innovation: What part of the US-based tech innovation couldn't have been (and actually were) achieved with open-s…

Can't edit anymore, but this little factoid:

> More importantly, even GDP per Capita wise: > > https://data.oecd.org/gdp/gross-domestic-product-gdp.htm > > EU per Capita GDP in 2022 is the same as USA 2016. Was the USA in 2016 struggling but now isn't? > > This is all bullshit. Economic output is more than high enough and rising steadily. The problem remains solely in the distribution of the ouptut.

The conclusion that economic output has been rising steadily, even in the last couple of years, is true. But the 2016 vs 2022 numbers are nominal, thus useless. There is a much more significant difference over time when working in PPP/Inflation adjusted numbers:

https://ourworldindata.org/grapher/gdp-per-capita-worldbank?...

The overall point holds though: The economic output of the EU is at 45K per capita today, the level of the US in 1997. The US was not a poor country in 1997. Germany is at the economic output per capita of 2009.

Did the US in 1997 suffer from the problem that it didn't produce enough economic output? Of course not.

And given that, adjusting for inflation, GDP per capita is at an all-time high, the conclusion that you're poorer because economic production is distributed to others is necessarily true. And it tracks, too. Corporate profits and the Dow Jones are not down. The already extremely wealthy have accumulated nearly two thirds of the new wealth being created since 2020:

https://www.oxfam.org/en/press-releases/richest-1-bag-nearly...

> Billionaire wealth surged in 2022 with rapidly rising food and energy profits. The report shows that 95 food and energy corporations have more than doubled their profits in 2022. They made $306 billion in windfall profits, and paid out $257 billion (84 percent) of that to rich shareholders. The Walton dynasty, which owns half of Walmart, received $8.5 billion over the last year. Indian billionaire Gautam Adani, owner of major energy corporations, has seen this wealth soar by $42 billion (46 percent) in 2022 alone.

Given these facts, if we have to accept lower economic production in the name of a fairer distribution of economic production, that seems more than acceptable to me.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#234

Earlier quoted context omitted.

It's definitely successful, and I was probably too harsh there. But I genuinely think the barriers that are left are damn near insurmountable. An awful lot has to change before a Greek tech workers can move to Sweden as easily as a Virginian can move to California.

I had two Greeks on my team at a large tech company in Sweden.

Do you think there are more Greeks in Sweden, or Virginians in Washington?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#235
post #142

Earlier quoted context omitted.

Europe is much much poorer than the US, actually. https://pbs.twimg.com/media/F3PGpsrWEAEiplB?format=jpg&name=...

EU GDP per capita 2022 is the same as US GDP per capita 2017. Unless you want to say that the US was much poorer in 2017 than it was in 2022 that's a fairly ridiculous statement. Also, the highest productivity places in the EU have much lower hours worked per capita than the US, with Germans on average working 25% less than Americans and the EU as a whole working 13% less than the US. https://data.oecd.org/emp/hours-…

> EU GDP per capita 2022 is the same as US GDP per capita 2017.

Can't edit anymore, but: That were nominal numbers, and thus useless. See here:

https://news.ycombinator.com/item?id=40673552

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#236
post #130

Earlier quoted context omitted.

IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.

I'm really interested, do you have a source for those percentages ? I tried to look for some service provider to publish this kind of metrics, but haven't found any.

Sorry, I can't. My employer doesn't publish this kind of metrics, either. What I posted was definitely just some very rough number off my brain.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#237
post #197

The market (and selling price) is reflecting the perceived value of nvidia's solution vs AMDs - comprehensively including tooling, software, TCO and managability. Also curious how many companies are dropping that much money on those kind of accelerators just to run 8x 7B param models in parallel... You're also talking about being able to train a 14B model on a single accelerator. I'd be curious to see how "full-accel…

mi300x win in some inference workloads, h100 win in training and some others inference workloads ( fp8 inference with tensorRT-llm , rocm is young but is growing fast ) in a single system ( 8x accelerators ) LLMs, mi300x has very competitive inference TCO vs h100 . also : AMD Instinct MI300X Offers The Best Price To Performance on GPT-4 According To Microsoft, Red Team On-Track For 100x Perf/Watt By 2027 https://wccf…

wccftech is an untrustworthy source.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#238

Earlier quoted context omitted.

Amazon doesn’t make anything anyone wants to buy? Not to be snarky, but if AWS counts as “nothing” I’d sure like a slice of nothing please.

AWS is a (collection of) service(s) which can scale and have low marginal cost of reproduction. Fire Tablets are a real thing you can buy, but they are shit so nobody does.

Sorry, are you suggesting AWS Services "aren't real things"?

If I pay for a database server in Virginia, how is that not real?

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#239

Earlier quoted context omitted.

I had two Greeks on my team at a large tech company in Sweden.

Do you think there are more Greeks in Sweden, or Virginians in Washington?

I just thought it was funny your example was my experience. But I also don't think it applies that much to tech workers, they dont need to learn swedish at all. Certainly not easy but not insurmountable as you say.

Re: AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

#240

Earlier quoted context omitted.

Maybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.

Apple doesn't have any hardware SIMD technology that I'm aware of. At best, Apple has Metal API which iOS video games use. I guess there's a level of SIMD-compute expertise here, but it'd take a lot of investment to turn that into a full scale GPU that tangos with Supercomputers. Software is a bit piece of the puzzle for sure, but Metal isn't ready for prime time. I'd say Apple is ahead of Intel (Intel keeps wasting…

Apple makes a better consumer GPU than AMD does.

M3 Max's GPU is significantly more efficient in perf/watt than RDNA3, already has better ray tracing performance, and is even faster than a 7900XT desktop GPU in Blender.[0]

[0]https://opendata.blender.org/benchmarks/query/?compute_type=...

Post reply on HN