Why is there a countdown?
It saves you from having to perform date-time calculations.
Kimi-K3 on HuggingFace
241–250 of 588 posts
Re: Kimi-K3 on HuggingFace
#242I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models
because imagine starting work every week and finding that your dumb ass CEO pivoted the company again and is ruining other peoples lives, and he then reorgs the management again so you now have your 5th leader this year.
Morale and momentum are huge things in companies, Zuck has been murdering both of those in Meta since... well naming it Meta.
Re: Kimi-K3 on HuggingFace
#243Earlier quoted context omitted.
Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K…
I find it difficult to always have one or more long-horizon tasks 'queued up' and ready to run... I find myself usually bottlenecked on design, review, or something similar that requires me being in the driver's seat. It's possible I could queue up a bunch of tasks, letting the LLM run off in multiple directions, but then I'd be less able to steer and course correct. Just my experience though, I'm still figuring thin…
In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide its thinking.
Re: Kimi-K3 on HuggingFace
#244Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture. I think it's very valuable to have a smaller model that doesn't h…
There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows. Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.
Isn’t that distillation ?
Re: Kimi-K3 on HuggingFace
#245Earlier quoted context omitted.
> Even if the output is like 5-6 tok/s On a 3T model I’d imagine you’d be closer to 0.05 tks
Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.
Re: Kimi-K3 on HuggingFace
#246Wait, so I can download it and run it locally now?? Wow...
But it probably won't work on my computer, right?
Re: Kimi-K3 on HuggingFace
#247This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…
Speaking of finetune, currently a common practice is LoRA over bnb 4-bit base model, but I think it's time to replace bnb with GGUF as the base model format. GGUF is actively supporting new model architectures and more aggressive quantizations. I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B…
I cannot be sure what the likes of Cursor have done, but I think it's incredibly unlikely that they have trained a QLoRA for Composer.
It's almost certainly full parameter post training of the original model weights.
Re: Kimi-K3 on HuggingFace
#248Earlier quoted context omitted.
Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K…
it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.
Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).
Re: Kimi-K3 on HuggingFace
#249Earlier quoted context omitted.
There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows. Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.
Isn’t that distillation ?
No. Distillation trains on a teach model's logits or output tokens.