...does Hugging Face have enough bandwidth to let people download en masse however much file size a 2 trillion parameters model is?
Kimi-K3 on HuggingFace
41–50 of 588 posts
Re: Kimi-K3 on HuggingFace
#42is there a realistic way to distill 2 consumer hardware friendly models with max ~200B and ~20B? Qwen did it, but would it be possible for 3rd parties (unsloth etc)?
Yeah, why not. Toughest part is running the hardware so you can create the traces for downstream training, but once over that hump, nothing would stop you from doing that no.
Re: Kimi-K3 on HuggingFace
#43This is historic. For the first time, an open-weights LLM is right at the top. We won't be able to run this ourselves, but many providers can.
> For the first time, an open-weights LLM is right at the top.
Hmm, not quite true, I think that honor, for better or worse, goes to OpenAI. When they released GPT2 (or GPT1 for that matter) is was quite literally the SOTA in the ecosystem when it was released.
Re: Kimi-K3 on HuggingFace
#44[flagged]
Re: Kimi-K3 on HuggingFace
#45I feel like most hardware to run LLMs on is shaped wrong for individuals.
It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards would be kinda useful (albeit NVLink or equivalent would need to be commonplace).
Obviously nobody is running Kimi K3 locally without an insanely beefy homelab and lots of money to burn, but running GLM 5.2 would be cool at like ~100 tokens per second for a single session and maybe ~60 tokens per second with N subagents.
How unfortunate.
Re: Kimi-K3 on HuggingFace
#46We already know that competition brought GLM 5.2 prices down roughly 45% since its release on June 16th (1.5 months ago), and the price downward slope is probably still going (I've been checking regularly and new providers keep fighting on price, I don't think prices have settled yet). For reference : https://openrouter.ai/z-ai/glm-5.2#providers I saw arguments like "Providers cannot price less than their costs" in o…
[dead]
Re: Kimi-K3 on HuggingFace
#47Re: Kimi-K3 on HuggingFace
#48This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…
> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing".
No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models.
Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
Re: Kimi-K3 on HuggingFace
#49Earlier quoted context omitted.
or 27/7 for the rest of the world
No, 27-7 for the rest of the world. The separator is often the only way to distinguish American notation from ISO, so please use a dash for dd-mm-yy and a forward slash for mm/dd/yy
Not really. Spain's traditional format is dd/mm/yyyy with slashes. This applies for a good chunk of Europe. Germany/Austria uses dot, I think nordic countries embraced the dash. While you might see more adoption in offical/digital contexts, I just double checked a few popular spanish websites, all slashes.
Re: Kimi-K3 on HuggingFace
#50This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…
It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes.
Huge price difference in what you can do with buying a used 4U rackmount server and putting 3TB of RAM in it (64GB DIMMs x quantity 32 in a quad socket xeon, you can see some benchmark prices on eBay for sets of 16 or 32 matched 64GB ECC DIMMs) for Now obviously, as of the time I write this, the full precision hasn't been released nor has anyone like unsloth run it through quantization yet to produce a "Q8" or "Q8-XL" variant of it. But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB.
I also predict that people who try to run it in Q4 and Q6 will get the worst of both worlds, less precision/lost knowledge but also not reliable output that comes out too slow. In my personal opinion if I'm going to deal with something that is smart but slow and running on limited budget hardware, I need it to be Q8.