Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

351–360 of 588 posts

Re: Kimi-K3 on HuggingFace

#351

Post by KerrAvon »

Davidzheng wrote: post #182
ttoinou wrote: post #144

Earlier quoted context omitted.

The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from 128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights

Not Kimi K3 large though

128 isn't enough, but if this report is accurate, the next Mac Studio could run it

https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-202...

Re: Kimi-K3 on HuggingFace

#352

Post by woctordho »

maxignol wrote: post #145
woctordho wrote: post #20

Earlier quoted context omitted.

Speaking of finetune, currently a common practice is LoRA over bnb 4-bit base model, but I think it's time to replace bnb with GGUF as the base model format. GGUF is actively supporting new model architectures and more aggressive quantizations. I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B…

I dont quite understand why GGUF is better optimized. Are the performances better for the same amount of VRAM ?

GGUF is at least better than bnb. From what I know, bnb does not yet find a way to quantize MoE with enough accuracy, and maintain the dequant-MoE kernels. In the age of Qwen 3.0, people tried to make some bnb '4-bit' quants of MoE models, but actually the MoE part is not quantized. It's a pity that even Unsloth gave up low-VRAM finetuning with MoE (although they're making their GGUFs for inference), and the world of local training looks stagnated for months.

GGUF is maintained by all the llama.cpp developers. There are many quantization formats and algorithms under this container format, some are optimized for MoE (such as APEX quant), some for CPU and some for GPU, some work surprisingly well below 4-bit (and even near 1-bit). It also supports recent architectures like linear attentions and mHC.

Re: Kimi-K3 on HuggingFace

#353

Post by lallysingh »

barbacoa wrote: post #285
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

They are saying that AMD's new Epyc Venice CPU has 16 memory channels allowing up to 1.6Tb/s of bandwidth. Which is higher bandwidth than most non-HBM GPUs. So full CPU local AI inference may become viable option in coming years.

This is essentially guaranteed. There are lots of useful smaller models that we should be able to run locally. Over time they'll be more and more capable and require less API usage.

Re: Kimi-K3 on HuggingFace

#354

Post by cmrdporcupine »

Curzel wrote: post #327
KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

RTX Spark does go up to 128gb No idea how it compares tho

I have one. The limitation (beyond total size of the memory) with the Spark is DDR5. "Real" inference hardware is HBM (high bandwidth memory) which is like 10x the performance.

So for prefill -- which is more about compute than bandwidth -- the Spark performs quite admirably. But on decode it's highly bandwidth constrained. Some smaller MoE models (e.g. Gemma4) can do 60-70 tok/second but anything dense, or anything that is actually filling up most of that 128GB is going to choke out around 15 tok/sec. Even at NVFP4.

My experiments with this are at https://github.com/rdaum/eider/

For my current work I get to log into trays on a real GB300. It's somewhat comical that NVIDIA is marketing the little baby on my shelf here as even in the same universe as that. Which is basically like having access to a super computer.

Re: Kimi-K3 on HuggingFace

#355

Post by apitman »

Iolaum wrote: post #116
walrus01 wrote: post #59

Earlier quoted context omitted.

I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rack…

There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1. Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it…

Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?

Re: Kimi-K3 on HuggingFace

#356

Post by NitpickLawyer »

achrono wrote: post #340
storus wrote: post #333

Getting 404 on the OP's link. Does it mean it got banned or self-censored in the meantime?

At the risk of sounding like a conspiracy theorist, this sounds like a great opportunity to make a statement . US or China, but likelier to be the former. Maybe Clem's on a call with the US government right now?

Open source teams have had access to the weights for at least a week now. vLLM folks expect full support on public release of the weights. Anything conspiratorial won't prevent the weights from leaking...

It's just launch day gremlins, like always.

Re: Kimi-K3 on HuggingFace

#357

Post by wren6991 »

qrios wrote: post #348
llm_nerd wrote: post #339

Earlier quoted context omitted.

I imagine it's just technical issues on the flip. It's also going to be interesting what happens to HF with loads of people downloading a many TB model. Even though almost no one has the capability to run it, it does seem like something to stash away in case it suddenly becomes unavailable due to government controls. FWIW, China is suddenly talking about model export controls. It was one thing to release also-ran mod…

> FWIW, China is suddenly talking about model export controls. ... Thanks for the hint. It makes sense, even if the main driver for releasing the models is to reduce the market size for the US companies. Do you have any sources? Edit: "market size"

The main source I'm aware of is this Reuters article: https://www.reuters.com/world/beijing-is-looking-curbing-ove...

However you should take it with a pinch of salt because IMO their direct quotations do not support their title.

Re: Kimi-K3 on HuggingFace

#358

Post by [deleted] »

codethief wrote: post #337
storus wrote: post #333

Getting 404 on the OP's link. Does it mean it got banned or self-censored in the meantime?

Until a few minutes ago there was a countdown page. (The weights haven't been released yet.) The countdown should be over in 19min, not sure why we're suddenly getting a 404.

[deleted]

Re: Kimi-K3 on HuggingFace

#360

Post by btown »

fooker wrote: post #63
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

One aspect of this is cyberattack proliferation by way of "Hey boss, I saw this TikTok that says if you let me invest [a tiny piece of the neighborhood's profit|our militia's budget] into some RAM, I could get a fully autonomous cyber operation up and running that pays for itself via ransomware etc. within weeks. You like it, we upgrade to something that can work even faster. We don't need the hacker guy from Swordfish with fifty monitors, we just need my cousin who likes building gaming PCs."

That's a world that I don't think we're ready for.

Post reply on HN