Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

thinkpol.ca

Post reply on HN
Print view View original

131–140 of 235 posts

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#131

Post by TeMPOraL »

ninjahawk1 wrote: post #61

At the current rate, open sourced models are expected to surpass cloud models within a couple years based on a study I read a couple days ago. Looking back at chatGPT and claude a couple years ago, very small Qwen models are basically equal in coding to what those cloud based models could do then. Also factoring in scaling laws, a 9b going to 18b is roughly a 40% increase, whereas 18b to 35b is 20%, I expect there wi…

That makes no sense, though, and reeks of extrapolating a trend way beyond the conditions in which it is valid.

The simple truth is, cloud models are always going to be strictly superior to open ones, simply because cloud model vendors can run those same open models too. And they still retain economies of scale and efficiency that operating large data centers full of specialized hardware, so at the very least they can always offer open models at price per token that's much less than anyone else's electricity bill for compute. But on top of that, they still have researchers working on models and everything around them; they can afford to put top engineers on keeping their harness always ahead of whatever is currently most popular on Github, etc.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#132

Post by Cookingboy »

pjerem wrote: post #102
0xbadcafebee wrote: post #43

These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…

The news is not in the way to compare models, it’s that Kimi K2.6 (and I’d add Deepseek v4 Pro) are more or less equivalent to Opus and that’s already pretty big. They are open source and cost waaaay less per token than American models. I’m using them right now on the $20 Ollama cloud plan and I can actually work with them on my side projects without reaching the limits too much. With Claude Pro $20 plan my usage can…

> for American economy.

There is more to American economy than big tech.

And that's precisely why this has started: https://www.wired.com/story/super-pac-backed-by-openai-and-p...

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#133

Post by [deleted] »

chvid wrote: post #129
pjerem wrote: post #102

Earlier quoted context omitted.

The news is not in the way to compare models, it’s that Kimi K2.6 (and I’d add Deepseek v4 Pro) are more or less equivalent to Opus and that’s already pretty big. They are open source and cost waaaay less per token than American models. I’m using them right now on the $20 Ollama cloud plan and I can actually work with them on my side projects without reaching the limits too much. With Claude Pro $20 plan my usage can…

Or maybe it is a pretty good thing for the American economy that you can get AI at cost rather than monopoly pricing. You know, for the rest of the economy that is not big tech.

[deleted]

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#134

Post by Galanwe »

tgv wrote: post #123
gertlabs wrote: post #34

I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's not surprising to see how well Kimi K2.6 performed. Based on our testing, for coding especially, Kimi is within statistical uncertainty of MiMo V2.5 Pro for top open weights model, and performs much better with tools…

This may be objectively scored, but it is not an indication of anyone's coding capabilities. This test measures which model almost accidentally came up with the best strategy (against other bots). This is not representative of coding. You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum, to get an idea which model is best at finding strategies involving an English dictionary.

> You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum

Would you? I am not very knowledgable on LLMs, but my understanding was that each query was essentially a stateless inference with previous input/output as context. In such a case, a single puzzle, yielding hundreds of queries, is essentially hundreds of paths dependent but individual tests?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#135

Post by joe_mamba »

Cookingboy wrote: post #132
pjerem wrote: post #102

Earlier quoted context omitted.

The news is not in the way to compare models, it’s that Kimi K2.6 (and I’d add Deepseek v4 Pro) are more or less equivalent to Opus and that’s already pretty big. They are open source and cost waaaay less per token than American models. I’m using them right now on the $20 Ollama cloud plan and I can actually work with them on my side projects without reaching the limits too much. With Claude Pro $20 plan my usage can…

> for American economy. There is more to American economy than big tech. And that's precisely why this has started: https://www.wired.com/story/super-pac-backed-by-openai-and-p...

>There is more to American economy than big tech.

Most of the stock market valuation is big-tech, and most of people's retirements are the stock market, so... if the AI bubble bursts a lot of the US will be affected.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#136

Post by joe_mamba »

sieve wrote: post #51

Kimi is really good. I have been using Sonnet and others (DeepSeek, ChatGPT, MiniMax, Qwen) for my compiler/vm project and the Claude Pro plan is mostly unusable for any serious coding effort. So I use it in chat mode in the browser where it cannot needlessly read your entire project, and use Kimi on the OpenCode Go plan with pi. Kimi consistently exceeded Sonnet on the C+Python project. Never had to worry about it d…

>the Claude Pro plan is mostly unusable for any serious coding effort

Why? Seems to go a giant the opinion of the masses who mostly use Claude Pro for serious coding.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#137

Post by gherkinnn »

I never looked in to the details of these benchmarks, I live with the assumptions that most benchmarks of any kind are gamed and useless.

What I do see in my own work and that of others around me, is that Claude consistently outperforms Gemini and to a lesser extent Codex.

With Claude eating tokens with declining return, concessions have to be made and Codex is a usable middle ground.

I use Kimi in Kagi's Assistant for non-code or generic programming questions and am quite happy with its no-bullshit responses.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#138

Post by biscuit1v9 »

Galanwe wrote: post #134
tgv wrote: post #123

Earlier quoted context omitted.

This may be objectively scored, but it is not an indication of anyone's coding capabilities. This test measures which model almost accidentally came up with the best strategy (against other bots). This is not representative of coding. You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum, to get an idea which model is best at finding strategies involving an English dictionary.

> You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum Would you? I am not very knowledgable on LLMs, but my understanding was that each query was essentially a stateless inference with previous input/output as context. In such a case, a single puzzle, yielding hundreds of queries, is essentially hundreds of paths dependent but individual tests?

Generally speaking, would you take a conclusion based only an event that happened once?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#139

Post by robbomacrae »

tgv wrote: post #123
gertlabs wrote: post #34

I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's not surprising to see how well Kimi K2.6 performed. Based on our testing, for coding especially, Kimi is within statistical uncertainty of MiMo V2.5 Pro for top open weights model, and performs much better with tools…

This may be objectively scored, but it is not an indication of anyone's coding capabilities. This test measures which model almost accidentally came up with the best strategy (against other bots). This is not representative of coding. You would need to test 100 or more of such puzzles, widely spread across the puzzle spectrum, to get an idea which model is best at finding strategies involving an English dictionary.

I don't think that is entirely fair.. I don't see them stating anywhere they are measuring coding capabilities... "Using complex games to probe real intelligence."

And this seems very much in line with the methodology in ARC-AGI-3.

The results here, in the OP article and in https://www.designarena.ai all tell a similar story: Kimi K2.6 is up and in the SOTA mix.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#140

Post by _pdp_ »

Kimi is capable model but it needs a very good harness. With a good harness it is a very capable model. But it can get into all kinds of issues (loops and such) something that frontier models do not.

As I said, you can blame the model, but it is nothing that the harness cannot take care of more deterministically.

Post reply on HN