Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

wafer.ai

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#71

Post by gpm »

anon373839 wrote: post #65
pocketarc wrote: post #63

Earlier quoted context omitted.

Your game example is a great one: If a game didn’t come with its 3D models, sound, level files, game data, or assets of any kind, but its engine was open source, I don’t think anyone would say “hey this game is open source”. They’d say the engine is, but that’s it. All of the above things are not code, but they are part of a game and yes, expected to be included in an open source game.

I think this misses the point I was trying to make. Both an open source game and an open weights LLM include the necessary assets. But people seem to be saying, “you need everything that was used to create the assets”. For game art, that would include sketches, prior art used as inspiration, project files that aren’t used by the game engine but were used during the design process, and so on - if you apply the same st…

Similar to this I've never seen a project request anyone license their github PR comments under any license whatsoever even though they're undoubtedly copyrighted and used in the creation of the actual source code.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#72

Post by HarHarVeryFunny »

kelnos wrote: post #10

I wish they wouldn't call them "open source models". They aren't open source. They didn't publish the training data. They didn't publish the tools they used to train the model. They published the weights. It's an "open weight model", a term that it seems nearly everyone has agreed is appropriate. Why is this company not using it?

> They didn't publish the tools they used to train the model.

That's not totally true - for example Ziphu (Z.ai, developers of GLM) have published their Slime RL-training framework, developed together with Tsinghua University.

https://github.com/THUDM/slime

Also, if you read Moonshot's Kimi 3 report, it does give a lot of architectural detail and details on training.

https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_repo...

Some discussion on this by HuggingFace here:

https://www.youtube.com/watch?v=MW8-kqd2SD8

Yes, we all realize that "open weights" is more accurate than "open source", and anyways the source code would not be very interesting - it's the training data and methods that mostly define these models.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#73

Post by croemer »

gpugreg wrote: post #37
Muromec wrote: post #35

Earlier quoted context omitted.

If you would tell people at the start of 20th century how much energy we consume, they may not believe you or think it is wasteful.

Looks like global energy consumption has risen by an order of magnitude from 1900 to 2000: https://www.encyclopedie-energie.org/en/world-energy-consump... Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.

Prices have dropped by a factor of around 6 in real terms since 1925 for US household electricity.

https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#74

Post by anon373839 »

croemer wrote: post #68
anon373839 wrote: post #62

Earlier quoted context omitted.

The antenna firmware is software. Model weights are essentially learned settings, and they are not software. Note also that software is copyrightable and model weights aren’t (in the US anyway).

Not sure why it matters whether weights are software or not. One can equally argue that firmware is just settings for the CPU. If weights aren't copyrightable then doesn't this argue against the idea of them being able to have the quality of open-sourceness? Meanwhile the training code could be copyrightable.

Well, my position is that the software that defines/runs the model is the open-source part.

The weights are the asset that makes it useful, but I don’t think 40T tokens of pre-training data should be required to call the model itself open source when you can inspect every line of code in the model without that, as well as instantiate and run the model with randomly initialized weights.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#75

Post by IshKebab »

inferencecoder wrote: post #45
logicallee wrote: post #42

Earlier quoted context omitted.

thanks for sharing. When I read your original comment, I was thinking you had just asked it to evaluate the article. (Like just "evaluate this article" or something.) I don't think anything anyone (or any AI) has ever written or published (including Sol itself) wouldn't be torn apart by the prompt you gave though.

I think it's fair to expect an extensive review of an article before publishing. Not everything has a set of serious flaws.

It doesn't look like it has found a set of serious flaws to me. Just nitpicking.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#76

Post by YetAnotherNick »

villgax wrote: post #18
inferencecoder wrote: post #9

Wafer is making themselves synonymous with slop in the inference space. Exaggerated unfair comparisons in all their results, twitter hype posts with alarm emojis etc. > $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200. This is not an accurate price comparison for real terms.

I went the gpus.io website & it’s $2.95/hr right now, this is like comparing MSRP to actual market price. B300s are in demand & hence cost more, but these lazy editors at wafer.ai can't be bothered to do TCO on actual ownership nor share code to replicate their setups. Instead just relying on current market prices to win one row, which isnt even about per/$ on actual MSRPs.

gpus.io shows tensorweave pricing at $2.95/hr. Tensorweave just shows "Talk to sales".

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#77

Post by ungovernableCat »

croes wrote: post #51
Almondsetat wrote: post #48

Earlier quoted context omitted.

Sorry, where exactly is "data" in "source"? I can understand not wanting to call it open source if they don't give you the algorithm and software used for training, but wanting the training data too? That's completely different

For LLMs the training data is the source of their weights. You can‘t reproduce the LLM without the same data

My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc

I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).

You'd likely get a model with very similar behaviour but the weights would be different.

Please someone correct me if I'm wrong.

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#78

Post by villgax »

throwa356262 wrote: post #31
villgax wrote: post #7

Lol, such a lazily written article by wafer.ai GPUs. 8× MI355X (TP8) B300 (TP8+DCP8) Decode tok/s per stream 118 tok/s 172 tok/s Peak aggregate. 952 tok/s 1,568 tok/s Peak aggregate per GPU 119 tok/s 196 tok/s On every row the B300 beat the MI355X The B200 is being forcefully compared against something which is not gonna fit within it's memory in a single node & not much details about multi-node interconnectivity, di…

Did you see this section? To the B200’s defence, its numbers are somewhat deflated by the fact that it pays a cross-node all-reduce on the decode critical path (RoCE v2 at ~195 Gb/s) — it’s the only config here that spans two nodes

No mention of Infiniband or SPX lol

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#79

Post by ungovernableCat »

HPsquared wrote: post #54
WithinReason wrote: post #46

Earlier quoted context omitted.

The weights are the modifiable representation. You modify them with gradient updates.

They are a cooked stew; the ingredients have already been chopped and mixed together. You can add things, yes, but you can't inspect the ingredients.

If we start with the same ingredients and independently make our stews they're likely still going to taste a bit different due to the non deterministic nature of the process. Isn't it similar with training LLMs at scale? Do you get identical weights if you do multiple runs?

Re: Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

#80

Post by fooker »

kelnos wrote: post #10

I wish they wouldn't call them "open source models". They aren't open source. They didn't publish the training data. They didn't publish the tools they used to train the model. They published the weights. It's an "open weight model", a term that it seems nearly everyone has agreed is appropriate. Why is this company not using it?

This trope was valid maybe in 2022.

Model training now is not a straight forwards process of input data -> run tools -> get model.

There's a whole lot of alchemy going on. We don't quite understand what works and what doesn't. Think of it like painting with water color and having to improvise very often.

The only advantage over water color is that we can revert to a working state.

Post reply on HN