Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion. Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory. What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings: • A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%. • Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against. • The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win. • A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%. There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people. The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs. Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.

Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through the numbers, flag what doesn't reconcile, draft the report — and there the two things that matter are that the model is good enough to trust with the judgement (K3 is, and it's the full 2.8T model, not a cut-down one) and that the data never leaves the machine.

I hate seeing comments like this on HN. We used to upvote “look at this crazy thing I did” work. Not everything has to make sense or be ground breaking.

It is cool that they got it to work at all.

This is difficult to read, maybe just link to a gist?

This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.