Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

91–100 of 588 posts

Re: Kimi-K3 on HuggingFace

#91

Post by walrus01 »

rhdunn wrote: post #84
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model? The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!] [!] With this and…

I assume you mean putting only the 32B active parameters on the GPU, and the rest on a bunch of regular server DRAM like on a 768GB to 1024GB RAM server?

Because Kimi K2.6 in Q4 is about 584GB GGUF size on disk and will use slightly more than that in RAM, Q8 is 595GB.

https://huggingface.co/unsloth/Kimi-K2.6-GGUF

Re: Kimi-K3 on HuggingFace

#92

Post by flyinglizard »

flaburgan wrote: post #78
Der_Einzige wrote: post #62

Earlier quoted context omitted.

Anyone who thinks that the labs are not profitable on per token API pricing is delusional and hilariously wrong.

It all depends if you count the fixed cost of training or not. And the cost of the hardware.

Or even the basis of the cost of hardware. There are lease deals, capacity traded for equity, various programs by Nvidia, there's absolutely massive depreciation, etc.

Re: Kimi-K3 on HuggingFace

#93

Post by Jarmsy »

egeozcan wrote: post #51
_zoltan_ wrote: post #32

Earlier quoted context omitted.

the only logical format. signed: a hungarian :)

For me, the only format that doesn't make sense is the MM/DD/YYYY, together with its rarely seen worse sibling, MM/DD/YY (07/27/26).

Let's just go with YM/DY/MD (27/26/07)

Re: Kimi-K3 on HuggingFace

#95

Post by leobg »

docheinestages wrote: post #67

Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture. I think it's very valuable to have a smaller model that doesn't h…

Why not generate an artificial dataset using commercial APIs and then finetune a small model on this data?

I’ve had success adapting even a 7B model for single-domain tasks that way, including reasoning and tool calling.

You can use an open model. The point is just to outsource the inference, so you don’t have to deal with running the larger model yourself.

Re: Kimi-K3 on HuggingFace

#96

Post by rvz »

This is (actually) AGI, that truly benefits all of humanity with zero gatekeeping.

Now the US government has 5 hours left to (attempt to) stop the release. (and save Anthropic)

Let competition run its course and the market (not government) determine the winners and losers.

Re: Kimi-K3 on HuggingFace

#97

Post by NitpickLawyer »

rhdunn wrote: post #84
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model? The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!] [!] With this and…

You're talking about running this "at home" for 1 user, using a mix of VRAM and RAM (total should be ~1.5TB). That's certainly possible. It'll be slow, especially prompt processing, but doable for single users.

But my comment on running it was more towards serving this profitably at scale. You get much better throughput / unit of compute if you load everything in VRAM and serve many requests at the same time. That's how all inference providers are doing it.

Re: Kimi-K3 on HuggingFace

#98

Post by jvuygbbkuurx »

embedding-shape wrote: post #68
amelius wrote: post #52

Earlier quoted context omitted.

> Without training cost you can infer only the marginal cost of serving this kind of models. Which is by far the most interesting number of the two. > Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?) If you get close in output quality, then does that matter?

> If you get close in output quality, then does that matter? When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?

That only matters if you are an investor not if you are a consumer.

Re: Kimi-K3 on HuggingFace

#100

Post by vorticalbox »

NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

I think cursor will likely do a grok fine tune rather than a kimi one for the next composer.

they noted in their blog post they didn't focus purely on coding for grok 4.5.

Post reply on HN