Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

aistack.imec-int.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#11

Post by infecto »

ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar.

Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#12

Post by michalpleban »

flifenstein wrote: post #2

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value fee…

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#13

Post by incognito124 »

ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#14

Post by deflator »

michalpleban wrote: post #12
flifenstein wrote: post #2

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value fee…

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run.

The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement.

Thank you for the article, though! Very informative.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#15

Post by jszymborski »

ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude.

Here are the tok/s I get:

- Gemma-4-26B-A4B (Q4_0) = 214 tok/s

- Gemma4-31B-QAT (Q4_0) = 58 tok/s

- Qwen3.6-35B-A3B (QB_0) = 30 tok/s

- Qwen3.6-27B (QB_0) = 9 tok/s

EDIT: Updated tok/s after updating llama.cpp

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#16

Post by Ilaurens »

jszymborski wrote: post #15
ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#17

Post by [deleted] »

Ilaurens wrote: post #16
jszymborski wrote: post #15

Earlier quoted context omitted.

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!

[deleted]

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#18

Post by [deleted] »

ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

[deleted]

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#19

Post by mft_ »

Ilaurens wrote: post #16
jszymborski wrote: post #15

Earlier quoted context omitted.

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!

Agree - I got 9.7 tok/s on an M1 Max with unsloth's gemma-4-31B-it-qat-UD-Q4_K_XL.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#20

Post by NitpickLawyer »

michalpleban wrote: post #12
flifenstein wrote: post #2

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value fee…

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

Kimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)
Post reply on HN