If the SWE Bench results are to be believed... this looks best in class right now for a local LLM. To be fair, show me the guy who is running this locally...
Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
21–30 of 194 posts
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#22Pelican on a bicycle result: https://simonwillison.net/2025/Jul/11/kimi-k2/
wow!
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#23If the SWE Bench results are to be believed... this looks best in class right now for a local LLM. To be fair, show me the guy who is running this locally...
It's challenging, but not impossible. With 2-bit quantisation, only about 250-ish gigabytes of RAM is required. It doesn't have to be VRAM either, and you can mix and match GPU+CPU inference.
In addition, some people on /r/localLlama are having success with streaming the weights off SSD storage at 1 token/second, which is about the rate I get for DeepSeek R1.
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#24Earlier quoted context omitted.
32B active parameters with a single shared expert.
This doesn’t change the VRAM usage, only the compute requirements.
It does not have to be VRAM, it could be system RAM, or weights streamed from SSD storage. Reportedly, the latter method achieves around 1 token per second on computers with 64 GB of system RAM.
R1 (and K2) is MoE, whereas Llama 3 is a dense model family. MoE actually makes these models practical to run on cheaper hardware. DeepSeek R1 is more comfortable for me than Llama 3 70B for exactly that reason - if it spills out of the GPU, you take a large performance hit.
If you need to spill into CPU inference, you really want to be multiplying a different set of 32B weights for every token compared to the same 70B (or more) instead, simply because the computation takes so long.
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#25Earlier quoted context omitted.
You can probably run this on CPU if you have a 4090D for prompt processing, since 1TB of DDR4 only comes out to around $600. For GPU inference at scale, I think token-level batching is used.
With 32B active parameters it would be ridiculously slow at generation.
DDR3 workstation here - R1 generates at 1 token per second. In practice, this means that for complex queries, the speed of replying is closer to an email response than a chat message, but this is acceptable to me for confidential queries or queries where I need the model to be steerable. I can always hit the R1 API from a provider instead, if I want to.
Given that R1 uses 37B active parameters (compared to 32B for K2), K2 should be slightly faster than that - around 1.15 tokens/second.
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#26I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models. From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own. Why not use existing "non-agentic" model and "orchestrate" them using LangChain…
> I'm sorry if my questions sound silly. Following AI world is like following JavaScript world.
You are more right than you could possibly imagine.
TL;DR: "agentic" just means "can call tools it's been given access to, autonomously, and then access the output" combined with an infinite loop in which the model runs over and over (compared to a one-off interaction like you'd see in ChatGPT). MCP is essentially one of the methods to expose the tools to the model.
Is this something the models could do for a long while with a wrapper? Yup. "Agentic" is the current term for it, that's all. There's some hype around "agentic AI" that's unwarranted, but part of the reason for the hype is that models have become better at tool calling and using data in their context since the early days.
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#27Earlier quoted context omitted.
This doesn’t change the VRAM usage, only the compute requirements.
It does not have to be VRAM, it could be system RAM, or weights streamed from SSD storage. Reportedly, the latter method achieves around 1 token per second on computers with 64 GB of system RAM. R1 (and K2) is MoE, whereas Llama 3 is a dense model family. MoE actually makes these models practical to run on cheaper hardware. DeepSeek R1 is more comfortable for me than Llama 3 70B for exactly that reason - if it spills…
The amount of people who will be using it at 1 token/sec because there's no better option, and have 64 GB of RAM, is vanishingly small.
IMHO it sets the local LLM community back when we lean on extreme quantization & streaming weights from disk to say something is possible*, because when people try it out, it turns out it's an awful experience.
* the implication being, anything is possible in that scenario
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#28How does it stack up against the new Grok 4 model?
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#29Would be hilarious if Zuck with his billion dollar poaching failed to beat budget Chinese models.
That reminds me of a thought I had about the poachings.
The poaching was probably more aimed at hamstringing Meta's competition.
Because the disruption caused by them leaving in droves is probably more severe than the benefits of having them on board. Unless they are gods, of course.
Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model
#30Would be hilarious if Zuck with his billion dollar poaching failed to beat budget Chinese models.
Wikipedia listed a FAIR alumni as cofounder for this "Moonshot AI". Make it funnier probably.