Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

thinkpol.ca

Post reply on HN
Print view View original

81–90 of 235 posts

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#81

Post by magicalhippo »

veber-alex wrote: post #48
gertlabs wrote: post #34

I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's not surprising to see how well Kimi K2.6 performed. Based on our testing, for coding especially, Kimi is within statistical uncertainty of MiMo V2.5 Pro for top open weights model, and performs much better with tools…

In my experience benchmarks are pretty meaningless. Not only is performance dependent on the language and tasks gives but also the prompts used and the expected results. In my own internal tests it was really hard to judge whether GPT 5.5 or Opus 4.7 is the better model. They have different styles and it's basically up to preference. There where even times where I gave the win to one model only to think about it more…

In addition, the harness around these models do a lot of work and changes the outcome significantly.

I just had an issue where Claude CLI with Opus 4.7 High could not figure out why my Blazor Server program was inert, buttons didn't do anything etc. After several rounds, I opened the web console and found that it failed to load blazor.js due to 404 on that file. I copied the error message to Claude CLI and after another several unproductive rounds I gave up.

I then moved the Codex, with ChatGTP 5.5 High. I gave it the code base, problem description and error codes. Unlike Claude CLI it spun up the project and used wget/curl to probe for blazor.js, and found indeed it was not served. It then did a lot more probing and some web searches and after a while found my project file was missing a setting. It added that and then probed to verify it worked.

So Codex fixed it in about 20 minutes without me laying hands on it (other than approve some program executions).

However, I'm not convinced this shows GPT 5.5 being that much better than Opus 4.7. It could very well be the harness around it, the system prompts used in the harness and tools available.

For reference this was me just trying to see how good the vibecoding experience is now, so was trying to do this as much hands-off as possible.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#83

Post by veber-alex »

magicalhippo wrote: post #81
veber-alex wrote: post #48

Earlier quoted context omitted.

In my experience benchmarks are pretty meaningless. Not only is performance dependent on the language and tasks gives but also the prompts used and the expected results. In my own internal tests it was really hard to judge whether GPT 5.5 or Opus 4.7 is the better model. They have different styles and it's basically up to preference. There where even times where I gave the win to one model only to think about it more…

In addition, the harness around these models do a lot of work and changes the outcome significantly. I just had an issue where Claude CLI with Opus 4.7 High could not figure out why my Blazor Server program was inert, buttons didn't do anything etc. After several rounds, I opened the web console and found that it failed to load blazor.js due to 404 on that file. I copied the error message to Claude CLI and after anot…

I actually noticed this too. GPT 5.5 is much more "hands on" with calling tools to debug issues and verify results. I did all my tests in Cursor but I don't know if they use a different system prompt for each model.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#84

Post by idonotknowwhy »

ljlolel wrote: post #79
0xbadcafebee wrote: post #43

These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…

[flagged]

So like Open Router?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#85

Post by Mashimo »

gertlabs wrote: post #34

I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's not surprising to see how well Kimi K2.6 performed. Based on our testing, for coding especially, Kimi is within statistical uncertainty of MiMo V2.5 Pro for top open weights model, and performs much better with tools…

Seems like in agentic work flow the qween flash and Deepseek flash models are quite good.

Fits with another comment from yesterday on here who said the flash models are just better at tool calling.

Planning with gpt55 and implementation with a flash model could be bang for the buck route.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#86

Post by Barbing »

slopinthebag wrote: post #45

Amazing. To me it feels like GLM 5.1, Kimi 2.6, DeepSeek 4 are all competitive both with each other and with the American models. Truly a great time to be alive. I would like to see more effort making the flash variants work for coding. They are super economical to use to brute force boilerplate and drudgery, and I wonder just how good they can be with the right harness, if it provides the right UX for the steering t…

>To me it feels like GLM 5.1, Kimi 2.6, DeepSeek 4 are all competitive both with each other and with the American models.

Yes, at least probably with each other

>Truly a great time to be alive.

Do wish outcome had been better for those whose work ended up in the training sets, wish competition could’ve found ways to agree on distillation practices, wish globally we’d planned as fast as we’re developing…

Tremendous excitement too

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#87

Post by walrus01 »

zozbot234 wrote: post #55
walrus01 wrote: post #49

People thinking to self-host Kimi K2.6 had better be prepared for how big it is. Q8 K XL quantization for instance is around 600GB on disk. I would bet about 700GB of VRAM needed. Quantizations lower than Q8 are probably worthless for quality. Or 2.05TB on disk for the full precision GGUF. https://huggingface.co/unsloth/Kimi-K2.6-GGUF If you can afford the hardware to run Kimi K2.6 at any decent speed for more than 1…

Kimi is a natively quantized model, the lossless full precision release is 595GB. Your own link mentions that.

the 'unsloth' link above is a 3rd party person that has quantized it to Q8, the original release is considerably larger in size than 600GB:

https://huggingface.co/moonshotai/Kimi-K2.6

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#88

Post by yorwba »

0xbadcafebee wrote: post #43

These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…

There are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they're just a fluke. If you fine-tune each model to achieve its full potential on the task you expect to be giving it, the rankings produced by different benchmarks even agree to a high degree: https://arxiv.org/abs/2507.05195

The author didn't do any of that. They ran each model once on each of 13 (so far) problems and then they chose to highlight the results for the 12th problem. That's not even p-hacking, because they didn't stop to think about p-values in the first place.

LLM quality is highly variable across runs, so running each model once tells you about as much about which one is better as flipping two coins once and having one come up heads and the other tails tells you about whether one of them is more biased than the other.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#90

Post by walrus01 »

zozbot234 wrote: post #70
CamperBob2 wrote: post #65

Earlier quoted context omitted.

So, realistically, $100K for an 8x RTX 6000 Pro system that can run it at a usable rate.

I think people will always disagree on what qualifies as a "usable rate". But keep in mind that practically no one sensible is running the latest Opus or GPT around the clock, especially not at sustainable, unsubsidized prices. With open-weights models it's easy to do that.

Also for people doing something medical, privacy or sensitive data related, there's an almost incalculable value (depending on industry niche) in having absolutely no external network traffic to any servers/systems you don't fully control.
Post reply on HN