Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

thinkpol.ca

Post reply on HN
Print view View original

161–170 of 235 posts

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#161

Post by adrian_b »

walrus01 wrote: post #87
zozbot234 wrote: post #55

Earlier quoted context omitted.

Kimi is a natively quantized model, the lossless full precision release is 595GB. Your own link mentions that.

the 'unsloth' link above is a 3rd party person that has quantized it to Q8, the original release is considerably larger in size than 600GB: https://huggingface.co/moonshotai/Kimi-K2.6

No.

I have downloaded Kimi-K2.6 (the original release).

  du -sh moonshotai/Kimi-K2.6 
  555G moonshotai/Kimi-K2.6

  du -s moonshotai/Kimi-K2.6 
  581255612 moonshotai/Kimi-K2.6
For comparison (sorted in decreasing sizes, 3 bigger models and 3 smaller models, all are recently launched):

  du -sh zai-org/GLM-5.1
  1.4T zai-org/GLM-5.1
  du -sh XiaomiMiMo/MiMo-V2.5-Pro 
  963G XiaomiMiMo/MiMo-V2.5-Pro
  du -sh deepseek-ai/DeepSeek-V4-Pro
  806G deepseek-ai/DeepSeek-V4-Pro

  du -sh XiaomiMiMo/MiMo-V2.5 
  295G XiaomiMiMo/MiMo-V2.5
  du -sh MiniMaxAI/MiniMax-M2.7
  215G MiniMaxAI/MiniMax-M2.7
  du -sh deepseek-ai/DeepSeek-V4-Flash
  149G deepseek-ai/DeepSeek-V4-Flash

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#162

Post by tgv »

Galanwe wrote: post #134
tgv wrote: post #123

Earlier quoted context omitted.

This may be objectively scored, but it is not an indication of anyone's coding capabilities. This test measures which model almost accidentally came up with the best strategy (against other bots). This is not representative of coding. You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum, to get an idea which model is best at finding strategies involving an English dictionary.

> You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum Would you? I am not very knowledgable on LLMs, but my understanding was that each query was essentially a stateless inference with previous input/output as context. In such a case, a single puzzle, yielding hundreds of queries, is essentially hundreds of paths dependent but individual tests?

From what I understood, it's a coding challenge: the models wrote a player for that specific word game. E.g. https://github.com/rayonnant-ai/aicc/blob/main/wordgempuzzle...

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#163

Post by tgv »

robbomacrae wrote: post #139
tgv wrote: post #123

Earlier quoted context omitted.

This may be objectively scored, but it is not an indication of anyone's coding capabilities. This test measures which model almost accidentally came up with the best strategy (against other bots). This is not representative of coding. You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum, to get an idea which model is best at finding strategies involving an English dictionary.

I don't think that is entirely fair.. I don't see them stating anywhere they are measuring coding capabilities... "Using complex games to probe real intelligence." And this seems very much in line with the methodology in ARC-AGI-3. The results here, in the OP article and in https://www.designarena.ai all tell a similar story: Kimi K2.6 is up and in the SOTA mix.

The task was writing a "bot" to play the game. The title is "Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge." How does that not imply measuring coding capabilities?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#164

Post by ode »

smj-edison wrote: post #69
magicalhippo wrote: post #24

Earlier quoted context omitted.

Currently it's not a huge difference given the subsidies of closed model subscriptions. Once that stops then yea it will be really nice to have open models as price competitors.

At least in my experience switching from Claude Pro ($20/month) to Kimi 2.6 through ollama (also $20/month), I was almost always hitting my usage limit with Sonnet 4.6, but with ollama I haven't hit my usage a single time.

How many t/ps do you get with Kimi on Ollama?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#165

Post by nmfisher »

ponyous wrote: post #74

Kimi is nowhere near GPT or Opus unfortunately. I really wish it was. I’m running evals where models have to generate code that produces 3D models and it’s obvious that it lacks spatial understanding and makes many more code errors before it succeeds. Maybe it’s better in one particular case here and there and I think this blog post is example of that.

Slightly OT, but after using DeepSeek V4 Pro for the last few weeks, I’ve found that it’s basically on par with Opus…except when it comes to driving Blender. This isn’t even a visual issue (DS isn’t multimodal), for whatever reason Opus just understands the Blender API a lot better.

There always seem to be pockets where closed frontier models perform slightly better.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#166

Post by adrian_b »

walrus01 wrote: post #49

People thinking to self-host Kimi K2.6 had better be prepared for how big it is. Q8 K XL quantization for instance is around 600GB on disk. I would bet about 700GB of VRAM needed. Quantizations lower than Q8 are probably worthless for quality. Or 2.05TB on disk for the full precision GGUF. https://huggingface.co/unsloth/Kimi-K2.6-GGUF If you can afford the hardware to run Kimi K2.6 at any decent speed for more than 1…

While most people would not be able to run Kimi K2.6 fast enough for a chat, as a coding assistant the low speed matters much less, especially when many tasks can be batched to progress during a single pass over the weights.

If you run it on your own hardware, you can run it 24/7 without worrying about token price or reaching the subscription limits and it is likely that you can do more work, even on much slower hardware. Customizing an open-source harness can also provide a much greater efficiency than something like Claude Code.

For any serious application, you might be more limited by your ability to review the code, than by hardware speed.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#167

Post by _pdp_ »

Imustaskforhelp wrote: post #146
_pdp_ wrote: post #144

Earlier quoted context omitted.

We have our own... so I was making the comment form our own experience working with the model. It is a lot trickier to use kimi compared to sonnet - hence why it seems that sonnet is more powerful while I think it is down to the harness.

How did you make your own harness, I am curious to know more about the building process and please feel free to share your harness. If someone were to not use your harness and rather use some stock harness though, what is the one that you would recommend? I am curious about that too.

I really don't mean to discourage anyone but I find that making your own harness is pretty complex process. We where very naive when we started, thinking this is just a loop, but then it turned out it has all kinds of nuances, edge-cases, things to be considered, etc.

As I am writing this I am fixing a bug in the harness that could cause infinite loops under some conditions.

For simple interactions looping over the the llm complete function is not really that difficult. Put some tools, write the loop and exit.

It starts getting more tricky when you need to detect cycling behaviour, when some tools might need to abort but do so gracefully, when you need to wait for something to complete while allowing other parts to continue, when you want to protect against too much usage after you hit certain thresholds, when you need to retry whatever, when you need to compact or truncate to maintain good context window and do so with the model in mind (kimi), when messages need to be structured in certain ways to handle various model capabilities, etc.

Pi.dev, Hermes and all the other I have seen do not do even half of the things I have enumerated above - not at least out of the box.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#169

Post by atraac »

joe_mamba wrote: post #136
sieve wrote: post #51

Kimi is really good. I have been using Sonnet and others (DeepSeek, ChatGPT, MiniMax, Qwen) for my compiler/vm project and the Claude Pro plan is mostly unusable for any serious coding effort. So I use it in chat mode in the browser where it cannot needlessly read your entire project, and use Kimi on the OpenCode Go plan with pi. Kimi consistently exceeded Sonnet on the C+Python project. Never had to worry about it d…

>the Claude Pro plan is mostly unusable for any serious coding effort Why? Seems to go a giant the opinion of the masses who mostly use Claude Pro for serious coding.

Anecdotal evidence but last week or two Claude changed something related to their quotas. I'm a Pro user(now Team Standard) and while I did quite a lot before with that subscription, past week the 5h quota quite literally lasts maybe 5 semi sized prompts. I don't "vibe" anything, I give it clearly defined tasks or things to debug/fix, nothing hardcore. I ran out of the quota every single day past week, often twice a day, this never happened before. It's rather unusable for actual professional usage now. I'm tempted to test Codex over next week to compare hence why we're waiting with going to Claude Max sub.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#170

Post by gigatexal »

atemerev wrote: post #143
joe_mamba wrote: post #135

Earlier quoted context omitted.

>There is more to American economy than big tech. Most of the stock market valuation is big-tech, and most of people's retirements are the stock market, so... if the AI bubble bursts a lot of the US will be affected.

I do not know why this is downvoted. This is true.

Agreed. I upvoted.
Post reply on HN