Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

aistack.imec-int.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#31

Post by NorwegianDude »

jszymborski wrote: post #15
ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for batch tasks.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#32

Post by jszymborski »

fhars wrote: post #28
jszymborski wrote: post #22

Earlier quoted context omitted.

Thanks for flagging, this is on my local rig and it's driving my display too. I'm curious now, will take a closer look. These are the tok/s as reported by LMStudio. EDIT: Updating llama.ccp gets me 58 tok/s on Gemma 31b

The Qwen-35B-A3B numbers are even weirder, did you drop a digit? I get half of that speed on a AMD Radeon RX 5500 XT (RADV NAVI14) (8192 MiB) (unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q6_K with q8_0 context).

No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading.

EDIT: I get 259 tok/s with the Q4_K_M quant

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#33

Post by [deleted] »

jszymborski wrote: post #32
fhars wrote: post #28

Earlier quoted context omitted.

The Qwen-35B-A3B numbers are even weirder, did you drop a digit? I get half of that speed on a AMD Radeon RX 5500 XT (RADV NAVI14) (8192 MiB) (unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q6_K with q8_0 context).

No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading. EDIT: I get 259 tok/s with the Q4_K_M quant

[deleted]

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#34

Post by serf »

flockonus wrote: post #30
jszymborski wrote: post #15

Earlier quoted context omitted.

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

To the models you say for coding, can you give an example of what you code? I find them insufficient for my projects (mid sized), but curious what people see working.

I use local models like qwen to do refactoring; the code and concept already exist, they need refining.

I use sota models for concept, implementation plans, and initial deployment.

I find that local models can perform at a very high level if given a very strict environment and the already bounded rules of an existing code base.

aye aye aye.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#36

Post by jckahn »

matheusmoreira wrote: post #35

The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.

Don't worry, you'll be able to rent your next computer: https://www.apple.com/newsroom/2026/07/apple-upgrade-launche...

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#37

Post by tuesdaynight »

matheusmoreira wrote: post #35

The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.

If the memory price don't go down anymore, we could say that we are already priced out.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#38

Post by pixelesque »

jszymborski wrote: post #32
fhars wrote: post #28

Earlier quoted context omitted.

The Qwen-35B-A3B numbers are even weirder, did you drop a digit? I get half of that speed on a AMD Radeon RX 5500 XT (RADV NAVI14) (8192 MiB) (unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q6_K with q8_0 context).

No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading. EDIT: I get 259 tok/s with the Q4_K_M quant

I assume you mean Q8_0, not QB_0? (was confused, as I've not obviously seen a QB_0 before and can't find one now).

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#39

Post by walrus01 »

ktosobcy wrote: post #5

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather…

Keep in mind that lm studio is just a GUI with llama.cpp/llama-server under the hood, so if you want to compile the latest llama-server and load your own choice of model, then connect to it with pi, opencode, etc or your own other choices of harness, that's also a popular option.

Things are moving fast enough these days that llama-server needs to be built from source every 4 or 5 days to keep up with model support and various tweaks in published quantized GGUF files.

Additionally there are a few different tweaks/branches of llama.cpp/llama-server that you can grab and compile to take advantage of changes people have made specific to discrete models and/or types of GPUs.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#40

Post by johndough »

NitpickLawyer wrote: post #20
michalpleban wrote: post #12

Earlier quoted context omitted.

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

Kimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)

HuggingFace used an NVFP4 quant of GLM-5.2 to investigate their latest hack, so that might also be worth a try:

https://huggingface.co/nvidia/GLM-5.2-NVFP4

Post reply on HN