Anecdotal, but having used Claude Code exclusively for the last several months, I was pleasantly surprised by how capable Pi + Kimi K2.6 is. It's also much faster (via OpenRouter), and at a fraction of the cost.
Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
91–100 of 235 posts
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#92These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…
There are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they're just a fluke. If you fine-tune each model to achieve its full potential on the task you expect to be giving it, the rankings produced by different benchmarks even agree to a high degree: https://arxiv.org/abs/2507.05195 The author d…
That's objective metrics. Not an objective way to compare, which is the selection of metrics to include.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#93What I would like to see is a comparison of how well the models work in long running conversations: * do they lie and gaslight * do they start breaking down on very long chats (forget old context, just get dumber) * do they constantly try to tell me how smart I am vs solving the problem (yes man) * do they follow conventions, parameters set out early in the prompts, or forget them * if they cant read a given file (li…
I think they all support branching if you use a agent like pi.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#94These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…
My theory is we will end up in a similar spot to hiring people. You can look at a CV (benchmarks) but you won't know for sure until you've worked with them for six months. We as an industry cannot determine if one software engineer is objectively better than another, on practically any dimension, so why do we think we can come to an objective ranking of models?
Not many things are as manifold broken as hiring these days. I hope we do not end up there.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#95Earlier quoted context omitted.
It absolutely does matter. The enshittification will go unnoticed at first but I'm already finding my favourite frontier models severely nerfed, doing incredibly dumb stuff they weren't in the past. We need open weight models to have a stable "platform" when we rely on them, which we do more and more.
Most people won't roll out their own K2 deployment across rented GPUs, so in that sense it doesn't matter that much, they'll be using a paid service which is just as much of a black box as Claude or ChatGPT. For example, on OpenRouter you can select a provider which state they use a given open model, but you have no idea what actually goes on behind the curtains, which quantization levels they use and so on. That sai…
> but you have no idea what actually goes on behind the curtains, which quantization levels they use and so on.
That would take something close to a global conspiracy of every technologist lying continuously to keep the tweaks secret. If necessary, I personally will rent some servers and run a vanilla Kimi K2.6 deployment for people to use at reasonable prices. I don't expect to ever make good on that threat because they are grim times indeed if I'm the first person doing something AI related, but the skill level required to load up a model behind an API is low.
So it isn't hard to see how there will be unadulterated Kimi models available and from there it is really, really straightfoward to tell if someone is quantising a model; just run some benchmarks against 2 different providers who both claim to serve the same thing. If one is quantising and another isn't there's a big difference in quality.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#96In a single challenge, measured by how performant the solution was. Kimi K2.6 is definitely a frontier-sized model, so on the one hand it's not that surprising it's up there with the closed frontier models. Being open is nice though, even though it doesn't matter that much for folks like me with a single consumer GPU.
> Being open is nice though, even though it doesn't matter that much for folks like me with a single consumer GPU. The value of open source is not that you will run it locally, it's that anyone can run it at all. Even if you can't afford to purchase the hardware to run large open source models, someone would, price it at half the cost of the closed source models and still make a profit. The only reason you are not se…
The reason nobody is pricing Kimi K2.6 at half the cost of the best closed source models is that there are too many providers of the same model, so the competition drives prices down and they have to charge far less than that. https://openrouter.ai/moonshotai/kimi-k2.6/providers
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#97Earlier quoted context omitted.
Kimi is a natively quantized model, the lossless full precision release is 595GB. Your own link mentions that.
the 'unsloth' link above is a 3rd party person that has quantized it to Q8, the original release is considerably larger in size than 600GB: https://huggingface.co/moonshotai/Kimi-K2.6
That page mentions that the model is natively INT4 for most of the params, and 600GB is in the ballpark of what's available there for download.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#98Earlier quoted context omitted.
There are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they're just a fluke. If you fine-tune each model to achieve its full potential on the task you expect to be giving it, the rankings produced by different benchmarks even agree to a high degree: https://arxiv.org/abs/2507.05195 The author d…
That's objective metrics. Not an objective way to compare, which is the selection of metrics to include.
That's exactly why there's a ton of different benchmarking suites used for evaluating hardware performance.
I reckon we'll have similar suites comparing different aspects of models.
And, at some point, we'll be dealing with models skewing results whenever they detect they're being benchmarked, like it happened before with hardware. Some say that's already happening with the pelican test.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#99These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and test…
My theory is we will end up in a similar spot to hiring people. You can look at a CV (benchmarks) but you won't know for sure until you've worked with them for six months. We as an industry cannot determine if one software engineer is objectively better than another, on practically any dimension, so why do we think we can come to an objective ranking of models?
You do not interview 1000 rounds on problems you're actually solving. If you did, hiring would be fine. Minus the social fit aspect, which isn't as relevant for a model.
Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge
#100Meanwhile, I can’t get kimi k2.6 to edit a heredoc in a shell script without it fucking it up.