Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

thinkpol.ca

Post reply on HN
Print view View original

61–70 of 235 posts

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#61

Post by ninjahawk1 »

At the current rate, open sourced models are expected to surpass cloud models within a couple years based on a study I read a couple days ago.

Looking back at chatGPT and claude a couple years ago, very small Qwen models are basically equal in coding to what those cloud based models could do then. Also factoring in scaling laws, a 9b going to 18b is roughly a 40% increase, whereas 18b to 35b is 20%, I expect there will be a change of at least price in cloud based models.

Adobe used to be $600 per month, then it became $20 when distribution scaled.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#62

Post by zozbot234 »

echelon wrote: post #6
magicalhippo wrote: post #2

In a single challenge, measured by how performant the solution was. Kimi K2.6 is definitely a frontier-sized model, so on the one hand it's not that surprising it's up there with the closed frontier models. Being open is nice though, even though it doesn't matter that much for folks like me with a single consumer GPU.

This is the future though. Open weights models that run on H200s provide far more opportunity to build products and real infrastructure around. You can always distill this for your little RTX at home. But models shaped for consumer hardware will never win wide adoption or remain competitive with frontier labs. This is something that _can_ compete. And it will both necessitate and inspire a new generation of open clou…

These large MoE models can work quite well on consumer or prosumer platforms, they'll just be slow, and you have to offset that by running them unattended around the clock. (Something that you can't really do with large SOTA models without spending way too much on tokens.) This actually works quite well for DeepSeek V4 series which has comparatively tiny KV-cache sizes so even a consumer platform can run big batches in parallel.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#63

Post by baxtr »

ninjahawk1 wrote: post #61

At the current rate, open sourced models are expected to surpass cloud models within a couple years based on a study I read a couple days ago. Looking back at chatGPT and claude a couple years ago, very small Qwen models are basically equal in coding to what those cloud based models could do then. Also factoring in scaling laws, a 9b going to 18b is roughly a 40% increase, whereas 18b to 35b is 20%, I expect there wi…

While this might be true I’m worried about the hardware side of things.

What if you have a good enough model but the cloud model providers are better in procuring hardware for interference?

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#64

Post by gertlabs »

veber-alex wrote: post #48
gertlabs wrote: post #34

I'm glad we're seeing a shift towards objectively scored tests. We've been doing this at scale at https://gertlabs.com/rankings , and although the author looks to be running unique one-off samples, it's not surprising to see how well Kimi K2.6 performed. Based on our testing, for coding especially, Kimi is within statistical uncertainty of MiMo V2.5 Pro for top open weights model, and performs much better with tools…

In my experience benchmarks are pretty meaningless. Not only is performance dependent on the language and tasks gives but also the prompts used and the expected results. In my own internal tests it was really hard to judge whether GPT 5.5 or Opus 4.7 is the better model. They have different styles and it's basically up to preference. There where even times where I gave the win to one model only to think about it more…

I think benchmarks are improving and will always have value, but it's the equivalent to someone's college and GPA for an entry level job application.

It's a strong signal for a job, but the soft skills are sometimes going to get Claude Opus 4.6 a job over smarter applicants. That's what we'd really like to measure objectively, and are actively working on.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#65

Post by CamperBob2 »

zozbot234 wrote: post #55
walrus01 wrote: post #49

People thinking to self-host Kimi K2.6 had better be prepared for how big it is. Q8 K XL quantization for instance is around 600GB on disk. I would bet about 700GB of VRAM needed. Quantizations lower than Q8 are probably worthless for quality. Or 2.05TB on disk for the full precision GGUF. https://huggingface.co/unsloth/Kimi-K2.6-GGUF If you can afford the hardware to run Kimi K2.6 at any decent speed for more than 1…

Kimi is a natively quantized model, the lossless full precision release is 595GB. Your own link mentions that.

So, realistically, $100K for an 8x RTX 6000 Pro system that can run it at a usable rate.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#68

Post by gleenn »

baxtr wrote: post #63
ninjahawk1 wrote: post #61

At the current rate, open sourced models are expected to surpass cloud models within a couple years based on a study I read a couple days ago. Looking back at chatGPT and claude a couple years ago, very small Qwen models are basically equal in coding to what those cloud based models could do then. Also factoring in scaling laws, a 9b going to 18b is roughly a 40% increase, whereas 18b to 35b is 20%, I expect there wi…

While this might be true I’m worried about the hardware side of things. What if you have a good enough model but the cloud model providers are better in procuring hardware for interference?

Local inference is definitely going to make more and more sense. Modern CPUs have all this amazing hardware well-optimized for inference purposes. I use a lot of web tools and see AI baked in and it feels weird. I want the smartness localized for speed and data security. I think and hope the industry points towards smart ai agents operating as locally as possible.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#69

Post by smj-edison »

magicalhippo wrote: post #24
DeathArrow wrote: post #17

Earlier quoted context omitted.

>Being open is nice though, even though it doesn't matter that much for folks like me with a single consumer GPU. Of course it matters because that makes coding plans much cheaper than those from Anthropic and OpenAI. For personal use I have coding plans with GLM 5.1, Kimi K2.6, MiniMax M2.7 and Xiaomi MiMo V2.5 Pro and I am getting a lot of bang for the buck.

Currently it's not a huge difference given the subsidies of closed model subscriptions. Once that stops then yea it will be really nice to have open models as price competitors.

At least in my experience switching from Claude Pro ($20/month) to Kimi 2.6 through ollama (also $20/month), I was almost always hitting my usage limit with Sonnet 4.6, but with ollama I haven't hit my usage a single time.

Re: Kimi K2.6 just beat Claude, GPT-5.5, and Gemini in a coding challenge

#70

Post by zozbot234 »

CamperBob2 wrote: post #65
zozbot234 wrote: post #55

Earlier quoted context omitted.

Kimi is a natively quantized model, the lossless full precision release is 595GB. Your own link mentions that.

So, realistically, $100K for an 8x RTX 6000 Pro system that can run it at a usable rate.

I think people will always disagree on what qualifies as a "usable rate". But keep in mind that practically no one sensible is running the latest Opus or GPT around the clock, especially not at sustainable, unsubsidized prices. With open-weights models it's easy to do that.
Post reply on HN