Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

161–170 of 588 posts

Re: Kimi-K3 on HuggingFace

#161

Post by zozbot234 »

petu wrote: post #134
rhdunn wrote: post #84

Earlier quoted context omitted.

If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model? The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!] [!] With this and…

You need whole weights in VRAM for optimal performance. Don't be confused by "experts" in the name -- you don't get to load static subset of experts and blast next 100 tokens with them. In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. > With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. Kimi K2.6 is released as INT4 already.…

> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available.

It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching.

You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability; though the obvious tradeoff is that having to store KV caches for the wider batches may leave you with less room to cache experts across layers and tokens.

(Plus if you're batching so widely that you end up loading essentially entire model layers, MTP then becomes applicable even for a MoE model. But this typically only applies if you're doing inference on a very large scale, or if your memory bandwidth is so scarce that you have to recover compute intensity by any means feasible.)

Re: Kimi-K3 on HuggingFace

#162

Post by ThunderSizzle »

acchow wrote: post #106
KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.

An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.

Re: Kimi-K3 on HuggingFace

#163

Post by [deleted] »

NitpickLawyer wrote: post #150
vb-8448 wrote: post #48

Earlier quoted context omitted.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario. Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve…

[deleted]

Re: Kimi-K3 on HuggingFace

#164

Post by Alpha3031 »

withinboredom wrote: post #118
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

That's very interesting. Does that mean you can reduce say, a 30B class Q8 from ~30 GB down to 10 GB or less?

Re: Kimi-K3 on HuggingFace

#165

Post by jeremyjh »

NitpickLawyer wrote: post #150
vb-8448 wrote: post #48

Earlier quoted context omitted.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario. Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve…

That isn't really relevant to GP's point. We still don't know what training costs because we don't know how much RL is done.

Re: Kimi-K3 on HuggingFace

#167

Post by criley2 »

vb-8448 wrote: post #48
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

>No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.

Are you talking about Kimi's training cost or the training cost of the model(s) that Kimi distilled?

Because Moonshot didn't even incur the majority of the training costs either

Re: Kimi-K3 on HuggingFace

#169

Post by TurdF3rguson »

FYI huggingface refers to the alien from the Aliens movies that we need to prevent from reaching earth at any cost because it means the end of civilization.

Just checking in because y'all sound good with that.

Post reply on HN