Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

aistack.imec-int.com

Post reply on HN
Print view View original

41–50 of 56 posts

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#41

Post by walrus01 »

michalpleban wrote: post #12
flifenstein wrote: post #2

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value fee…

Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.

So the 'unsloth' Kimi K3 Q8 quantization is about 1.56TB which is close to the original safetensors file in size, and should be nearly lossless. Realistically with enough context you'd need 2TB total RAM whether it's run on a CPU-only platform (like a dual socket xeon with 1024GB connected to each CPU) or some combination of CPU-connected-RAM and GPUs.

Unsloth Q4 is 1.51TB which doesn't really help much and isn't likely to be any different in performance.

Unsloth Q2-K-XL is 861GB and could possibly fit on 1TB resources but I would want to see a very thorough series of tests to see how much knowledge and capability is lost between it and the full thing.

Using an example from the much smaller gemma 4 31B because it's a decent set of charts I could find quickly, I don't know if anyone has published KL divergence charts for Q4 vs Q4 vs Q8 of Kimi K3:

https://localbench.substack.com/p/gemma-4-31b-gguf-kl-diverg...

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#42

Post by walrus01 »

Quoting from the article:

> In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.8 (62.5% for both).

I think there's some less tangible advantages to self-hosting something on the scale of Kimi K3 that can't be quantified in a specific number like token/s or percentage of problems solved. Such as:

a) data privacy/sovereignty from a wide range of possible perspectives, from medical to personal to "we can't have our data go to the USA" for some Canadians and Europeans.

b) being able to give it information security/network security tasks and red team scenarios without triggering claude or openai refusals.

c) being able to give it information security/network security tasks with zero risk of getting your account banned or investigated by anthropic or openai.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#43

Post by matheusmoreira »

tuesdaynight wrote: post #37
matheusmoreira wrote: post #35

The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.

If the memory price don't go down anymore, we could say that we are already priced out.

The memory prices put computers into the same category as cars. Expensive but still realistic.

It's these 200kUSD+ nodes with a dozen or more data center class GPUs that are killing the self-hosting dream. The cheapest nodes are is in the same price category as literal houses.

Either we get some competition in this space, or computing will go back to their roots as ivory tower big iron.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#44

Post by matheusmoreira »

jckahn wrote: post #36
matheusmoreira wrote: post #35

The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.

Don't worry, you'll be able to rent your next computer: https://www.apple.com/newsroom/2026/07/apple-upgrade-launche...

It's not like people ever owned iPhones to begin with. They're Apple's computers, Apple's just generously allowing their customers to use them, and only on their terms. A monthly iPhone subscription is just the end game.

You will nothing, and you'll be happy.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#45

Post by personjerry »

walrus01 wrote: post #42

Quoting from the article: > In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.…

why are information security / network security tasks bannable?

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#46

Post by walrus01 »

personjerry wrote: post #45
walrus01 wrote: post #42

Quoting from the article: > In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.…

why are information security / network security tasks bannable?

You'd have to ask Anthropic or OpenAI, but I've seen probably more than a dozen reports of it happening. You also risk getting your account banned for discussing certain risky things in chemistry, biochemistry, weapons, conflict, terrorism/history of guerilla warfare.

I imagine that within Anthropic and OpenAI there is a huge data set of "big no no!" topics that they maintain and continually add to. For alignment training and for more blunt purposes. For instance ChatGPT 3.5 could be instructed, as satire, to write a press release in the style of the English language propganda arm of ISIS but they put a stop to that fairly quick.

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#48

Post by StopTencent »

serf wrote: post #34
flockonus wrote: post #30

Earlier quoted context omitted.

To the models you say for coding, can you give an example of what you code? I find them insufficient for my projects (mid sized), but curious what people see working.

I use local models like qwen to do refactoring; the code and concept already exist, they need refining. I use sota models for concept, implementation plans, and initial deployment. I find that local models can perform at a very high level if given a very strict environment and the already bounded rules of an existing code base. aye aye aye.

[dead]

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#49

Post by jszymborski »

pixelesque wrote: post #38
jszymborski wrote: post #32

Earlier quoted context omitted.

No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading. EDIT: I get 259 tok/s with the Q4_K_M quant

I assume you mean Q8_0, not QB_0? (was confused, as I've not obviously seen a QB_0 before and can't find one now).

bahaha, LMStudio's badge looks a lot like a B but the tooltip says it is indeed an 8 (I just assumed it was B for "byte", but that's a poor initialism considering "bit").

https://dl.jszym.com/share/boards/pictures/Screenshot_202607...

Re: Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

#50

Post by jszymborski »

NorwegianDude wrote: post #31
jszymborski wrote: post #15

Earlier quoted context omitted.

On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s…

You can get easily over 100 tok/s on the Gemma 4 31B QAT if you enable MTP. Same goes for the 27B. I'm getting 880 tok/s of throughout on a single RTX 5090 for batch tasks.

I've switched over to unsloth studio and am hitting those speeds now, thanks for the tip.
Post reply on HN