The benchmarks on all these models are meaningless
Why and what would a good benchmark look like?
30 people trying out all models on the list for their use case for a week and then checking what they're still using a month after.
Kimi K2.5 Technical Report [pdf]
41–50 of 146 posts
Re: Kimi K2.5 Technical Report [pdf]
#42Earlier quoted context omitted.
Just curious - how does it compare to GLM 4.7? Ever since they gave the $28/year deal, I've been using it for personal projects and am very happy with it (via opencode). https://z.ai/subscribe
From what people say, it's better than GLM 4.7 (and I guess DeepSeek 3.2) But it's also like... 10x the price per output token on any of the providers I've looked at. I don't feel it's 10x the value. It's still much cheaper than paying by the token for Sonnet or Opus, but if you have a subscribed plan from the Big 3 (OpenAI, Anthropic, Google) it's much better value for $$. Comes down to ethical or openness reasons t…
Exactly. For the price it has to beat Claude and GPT, unless you have budget for both. I just let GLM solve whatever it can and reserve my Claude budget for the rest.
Re: Kimi K2.5 Technical Report [pdf]
#43Earlier quoted context omitted.
Can you share how you're running it?
Running it via https://platform.moonshot.ai -- using OpenCode. They have super cheap monthly plans at kimi.com too, but I'm not using it because I already have codex and claude monthly plans.
Where? https://www.kimi.com/code starts at $19/month, which is same as the big boys.
Re: Kimi K2.5 Technical Report [pdf]
#44Earlier quoted context omitted.
You can plug another model in place of Anthropic ones in Claude Code.
That tends to work quite poorly because Claude Code does not use standard completions APIs. I tried it with Kimi, using litellm[proxy], and it failed in too many places.
It worked very well for me using qwen3 coder behind a litellm. Most other models just fail in weird ways though.
Re: Kimi K2.5 Technical Report [pdf]
#45Earlier quoted context omitted.
To save everyone a click > The 1.8-bit (UD-TQ1_0) quant will run on a single 24GB GPU if you offload all MoE layers to system RAM (or a fast SSD). With ~256GB RAM, expect ~10 tokens/s. The full Kimi K2.5 model is 630GB and typically requires at least 4× H200 GPUs. If the model fits, you will get >40 tokens/s when using a B200. To run the model in near full precision, you can use the 4-bit or 5-bit quants. You can use…
I'm running the Q4_K_M quant on a xeon with 7x A4000s and I'm getting about 8 tok/s with small context (16k). I need to do more tuning, I think I can get more out of it, but it's never gonna be fast on this suboptimal machine.
The pitiful state of GPUs. $10K for a sloth with no memory.
Re: Kimi K2.5 Technical Report [pdf]
#46I really like the agent swarm thing, is it possible to use that functionality with OpenCode or is that a Kimi CLI specific thing? Does the agent need to be aware of the capability?
Has anyone tried it and decided it's worth the cost; I've heard it's even more profligate with tokens?
Re: Kimi K2.5 Technical Report [pdf]
#47I wonder how K2.5 + OpenCode compares to Opus with CC. If it is close I would let go of my subscription, as probably a lot of people.
It is not opus. It is good, works really fast and suprisingly through about its decisions. However I've seen it hallucinate things. Just today I asked for a code review and it flagged a method that can be `static`. The problem is it was already static. That kind of stuff never happens with Opus 4.5 as far as I can tell. Also, in an opencode Plan mode (read only). It generated a plan and instead of presenting it and s…
Some models have a mind of their own. I keep them on a leash with `permission` blocks in OC -- especially for rm/mv/git.
Re: Kimi K2.5 Technical Report [pdf]
#48Earlier quoted context omitted.
Out of curiosity, what kind of specs do you have (GPU / RAM)? I saw the requirements and it's a beyond my budget so I am "stuck" with smaller Qwen coders.
I'm not running it locally (it's gigantic!) I'm using the API at https://platform.moonshot.ai
How long until this can be run on consumer grade hardware or a domestic electricity supply I wonder.
Anyone have a projection?
Re: Kimi K2.5 Technical Report [pdf]
#49Earlier quoted context omitted.
You can plug another model in place of Anthropic ones in Claude Code.
That tends to work quite poorly because Claude Code does not use standard completions APIs. I tried it with Kimi, using litellm[proxy], and it failed in too many places.
opencode is a good alternative that doesnt flake out in this way.
Re: Kimi K2.5 Technical Report [pdf]
#50Earlier quoted context omitted.
A lot better in my experience. M2.1 to me feels between haiku and sonnet. K2.5 feels close to opus. That's based on my testing of removing some code and getting it to reimplement based on tests. Also the design/spec writing feels great. You can still test k2.5 for free in OpenCode today.
Well, Minimax was the equivalent of Sonnet in my testing. If Kimi approach Opus, that would be great.
Kimi K2.5 approaches Sonnet as well from what I can tell, it's just slower to get to the result.