Thanks for sharing, tito! I figured it was time I chimed in properly, as the lead on this project.
Quick mission statement, since it explains everything below: Deltafin runs the full, never-pruned, 2.8-trillion-parameter Kimi K3... all 16 experts, every token, no shortcuts, nothing re-encoded down to "close enough." Speed is not allowed to come from quality.
A few similar K3 projects have shown up claiming to be the same thing, just faster. But they all get there by shrinking the model - quantizing it down, trading fidelity for speed, calling the results close enough. Those are great projects, and have real value - but they're very different projects. Deltafin's purpose remains the real full K3, unaltered and uncut, as fast* as possible.
* it's not fast
Next, an update since this was posted five days ago. Thanks to everyone who filed issues, including a couple of you running M3 Ultra and M5 Max boxes, which helped a lot!
Since this was posted, throughput on the M1 Max laptop went from 0.0141 tok/s to 0.2847 tok/s - about 20x, and still climbing. Most of that came from routine hardening: fixing a storage race, a rust/c++ rewrite, and tolerating a float32 rounding drift that looked like a routing bug until it wasn't.
None of that is a moonshot (get it?). It's the boring, necessary work of making something this huge survive on consumer hardware.
Now one huge thing to get out of the way, because this is about half the comments: is this a viable business product, the kind of thing that makes people in suits happy because it helps them sell widgets? No. My interest in computing goes beyond that, and I suspect a lot of yours does too. I'd like to think that's still what hacker news is largely about.
A research project usually isn't the useful thing itself. It's what falls out of chasing it.
It is genuinely hard to squeeze even a small speed gain out of a model this size on a machine this small. Yes, an M1 Max is a good laptop. But Kimi's own recommended setup is 16 nodes at ~300GB of VRAM each. That's $2,000,000 of hardware. Next to that, this laptop is an ant lifting a car. And that is the point of this project - the insane challenge.
My hope is that if we push as hard as we can to get the largest public model ever released running on the smallest machine that can plausibly hold it, some of what falls out along the way could be useful elsewhere... maybe to smaller models, on smaller machines, running faster than they do today.
But first I wanted to find out whether this was even possible at all. K3 is the largest public model that exists. Nobody had run something this size on a laptop before. The closest comparable projects like colibri and DwarfStar ran quantized models at half this size. Turns out: yes, it's doable, but slow af.
But I've spent a long time frustrated with the quality ceiling on local models, and quantization is a big part of that ceiling. The hallucinations. The reasoning that almost holds together, but then doesn't.
So I'm done compromising on that axis. This isn't going to be another quantized, slightly-worse copy of the real model wearing its name.
Maybe no techniques we’re using are groundbreaking at all in the end. Bummer. But, at least I tried. Try and fail at life, people. That’s okay. Just try, and at least you’ll have lived a little. It's a decent way to spend the time you get.
If people want to keep following this, I'll post updates here instead of just on Reddit like I have been.