Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

twitter.com

Post reply on HN
Print view View original

11–20 of 194 posts

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#11

Post by maven29 »

JustFinishedBSG wrote: post #10
maven29 wrote: post #9

Earlier quoted context omitted.

32B active parameters with a single shared expert.

This doesn’t change the VRAM usage, only the compute requirements.

You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600.

For GPU inference at scale, I think token-level batching is used.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#12

Post by t1amat »

maven29 wrote: post #11
JustFinishedBSG wrote: post #10

Earlier quoted context omitted.

This doesn’t change the VRAM usage, only the compute requirements.

You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600. For GPU inference at scale, I think token-level batching is used.

With 32B active parameters it would be ridiculously slow at generation.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#14

Post by zackangelo »

maven29 wrote: post #11
JustFinishedBSG wrote: post #10

Earlier quoted context omitted.

This doesn’t change the VRAM usage, only the compute requirements.

You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600. For GPU inference at scale, I think token-level batching is used.

Typically a combination of expert level parallelism and tensor level parallelism is used.

For the big MLP tensors they would be split across GPUs in a cluster. Then for the MoE parts you would spread the experts across the GPUs and route to them based on which experts are active (there would likely be more than one if the batch size is > 1).

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#15

Post by wiradikusuma »

I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models.

From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own.

Why not use existing "non-agentic" model and "orchestrate" them using LangChain, MCP etc? Why create a new breed of model?

I'm sorry if my questions sound silly. Following AI world is like following JavaScript world.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#16

Post by ozten »

wiradikusuma wrote: post #15

I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models. From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own. Why not use existing "non-agentic" model and "orchestrate" them using LangChain…

It is not a silly question. The various flavors of LLM have issues with reliability. In software we expect five 9s, LLMs aren't even a one 9. Early on it was reliability of them writing JSON output. Then instruction following. Then tool use. Now it's "computer use" and orchestration.

Creating models for this specific problem domain will have a better chance at reliability, which is not a solved problem.

Jules is the gemini coder that links to github. Half the time it doesn't create a pull request and forgets and assumes I'll do some testing or something. It's wild.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#19

Post by simonw »

wiradikusuma wrote: post #15

I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models. From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own. Why not use existing "non-agentic" model and "orchestrate" them using LangChain…

"Agentic" and "agent" can mean pretty much anything, there are a ton of different definitions out there.

When an LLM says it's "agentic" it usually means that it's been optimized for tool use. Pretty much all the big models (and most of the small ones) are designed for tool use these days, it's an incredibly valuable feature for a model to offer.

I don't think this new model is any more "agentic" than o3, o4-mini, Gemini 2.5 or Claude 4. All of those models are trained for tools, all of them are very competent at running tool calls in a loop to try to achieve a goal they have been given.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#20

Post by dcre »

wiradikusuma wrote: post #15

I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models. From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own. Why not use existing "non-agentic" model and "orchestrate" them using LangChain…

Reasonable question, simple answer: "New breed of model" is overstating it — all these models for years have been fine-tuned using reinforcement learning on a variety of tasks, it's just that the set of tasks (and maybe the amount of RL) has changed over time to include more tool use tasks, and this has made them much, much better at the latter. The explosion of tools like Claude Code this year is driven by the models just being more effective at it. The orchestration external to the model you mention is what people did before this year and it did not work as well.
Post reply on HN