Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

wafer.ai

Post reply on HN
Print view View original

101–110 of 119 posts

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#101

Post by logicallee »

springtimesun wrote: post #100
serf wrote: post #95

Earlier quoted context omitted.

>and its will confidently tell you it’s found the best setup when it’s only turned a few knobs. on one hand , yeah : a model being more aggressive towards exploration of the decision space is usually a good thing. on the other hand : I think that it's up to the operator to set rigid test criteria to make these things actually work well in a repeatable fashion. so in other words, i'm glad claude is doing a better job…

It’s not a complaint, only an observation. Operator aside, to observe that the models behave differently shouldn’t be controversial, should it? It seems like you read my comment as dissing Kimi. Kimi is in regular rotation for me. Some tasks that Claude used to own go to Kimi first now. They are just different tools with their own strengths and weaknesses.

Thanks for sharing. What kind of tasks do you give Kimi rather than Claude?

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#102

Post by BoorishBears »

inferencecoder wrote: post #9

Wafer is making themselves synonymous with slop in the inference space. Exaggerated unfair comparisons in all their results, twitter hype posts with alarm emojis etc. > $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200. This is not an accurate price comparison for real terms.

Most discourse around GPU prices is nonsense right now. Some people using Spot prices for providers who won't have Spot capacity, some people using hourly rates for instances that are never in stock, some people ignoring commitment discounts.

Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#103

Post by inferencecoder »

IshKebab wrote: post #75
inferencecoder wrote: post #45

Earlier quoted context omitted.

I think it's fair to expect an extensive review of an article before publishing. Not everything has a set of serious flaws.

It doesn't look like it has found a set of serious flaws to me. Just nitpicking.

To begin with, as I pointed out separately, the price comparison that makes the headline is absurd. No one is easily getting the MI355X's at $2.5/hr. And then the eval is unreplicable and poorly defined.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#104

Post by inferencecoder »

BoorishBears wrote: post #102
inferencecoder wrote: post #9

Wafer is making themselves synonymous with slop in the inference space. Exaggerated unfair comparisons in all their results, twitter hype posts with alarm emojis etc. > $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200. This is not an accurate price comparison for real terms.

Most discourse around GPU prices is nonsense right now. Some people using Spot prices for providers who won't have Spot capacity, some people using hourly rates for instances that are never in stock, some people ignoring commitment discounts. Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation

> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation

The GPU price discourse is absurd, but many are serving models on single node setups when the model fits

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#105

Post by pertymcpert »

logicallee wrote: post #3

This part sounds like AI assisted setting this up and benchmarking it: >The fix was trivially simple: zero-pad the head count 12→16, run the fast kernel, and extract the real 12 heads from the output. I've recently used a frontier AI (ChatGPT 5.6 Sol on ultra) to set up a much smaller local model, and the performance optimizations it introduced left the model totally incoherent. (The model just repeats a single chara…

You can tell because instead of making an arrow like this: -> it uses a Unicode symbol instead.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#107

Post by HarHarVeryFunny »

croemer wrote: post #26
WithinReason wrote: post #20

Earlier quoted context omitted.

If they open sourced the training data and code but you had to train the network yourself, would that be open source?

Yes, obviously.

No, obviously!

These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals).

The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.

You don't need the source code - just read their architecture docs and implement it yourself.

The Chinese have actually been very open about training, starting perhaps with the DeepSeek-R1 paper which told the world in detail how to train a reasoning model. The Kimi 3 paper also gives a lot of training details.

There really isn't much of a comparison to be had between building a traditional software project where all you need is the (maybe open source) source code and the Makefile that automates the build process, and building a machine learning system where it's primarily about data not source code, and even with a road map of what may be very complex training (cf build) process, you'd probably still have a hard time building it since AFAIK the training process may still involve expert knowledge and intervention - I don't think training has been reduced to a hands-off "Makefile" or build script.

So, basically lack of source code is the least of the issues in being able to build one of these models - it's mostly the training data and training processes/expertise that you would need.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#108

Post by throw10920 »

HarHarVeryFunny wrote: post #107
croemer wrote: post #26

Earlier quoted context omitted.

Yes, obviously.

No, obviously! These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals). The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data. You don…

> The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data.

...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the ones being used.

The reason that the training data is missing is that they're trained on a large amount of American copyrighted data and distilled on American models, which is where a lot of their performance comes from.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#109

Post by HarHarVeryFunny »

throw10920 wrote: post #108
HarHarVeryFunny wrote: post #107

Earlier quoted context omitted.

No, obviously! These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals). The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data. You don…

> The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data. ...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the…

You can replicate the architectural innovations, and try them for yourself with your own dataset. It seems some of them are certainly being used by western companies, such as DeepSeek Sparse Attention, now supported by NVIDIA cuDNN.

Ditto for training algorithms and procedures such as Slime or DeepSeek's details instructions on how to build a reasoning model.

This is the exact value of openly shared details - others CAN copy and try them and modify them themselves.

Yes, the training data specifically has not been released for any model, American or Chinese, but that doesn't detract from what has been shared, and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing - because they are all using data from sources they don't want you to know about, and at the end of the day the data is the closest thing any of them do have to a moat.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#110

Post by BoorishBears »

inferencecoder wrote: post #104
BoorishBears wrote: post #102

Earlier quoted context omitted.

Most discourse around GPU prices is nonsense right now. Some people using Spot prices for providers who won't have Spot capacity, some people using hourly rates for instances that are never in stock, some people ignoring commitment discounts. Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation

> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation The GPU price discourse is absurd, but many are serving models on single node setups when the model fits

You can't beat current API pricing running Kimi K3 a single node, so as I said anyone serious is not doing that for Kimi K3.

Not sure how you're getting "no one serves models on a single node" out of that.

Post reply on HN