Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

21–30 of 157 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#21

Post by springtimesun »

pavlov wrote: post #10
Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.

But, Kimi thought for 36k tokens before writing it.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#24

Post by hypfer »

anamexis wrote: post #15
woadwarrior01 wrote: post #12

Earlier quoted context omitted.

That's because it's copy pasted from a coding agent.

It's difficult to read because it doesn't have line breaks.

And full of obvious markers of LLM-generated text.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#25

Post by copperx »

Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

The hyphenated terms get worse and worse as you keep reading. Just kill me now.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#26

Post by Argonautlabs »

sampullman wrote: post #8
Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

This is difficult to read, maybe just link to a gist?

Thank you! Here is the short version:

  Kimi K3, 2.78T parameters, ~1.45 TB of MXFP4 experts streamed from four SSDs on an M5 Max / 128 GB. 1.00 tok/s steady over 512 tokens, 1.13 over 128, ~6.3 min to first token on a 512-token prompt. Output token-identical drafter on/off on a given drive layout; the int8 trunk is non-weight-exact per upstream.
  
  The useful bits: one drive gives ≈52% of four, two ≈73%, three ≈90%; and prefill is slow because of ~9 TB of reads for a 1.4 TB model — a scheduling bug with a planned fix.
  
  README with per-run logs: github.com/argonautlabsai/deltafin — a fork of gavamedia/deltafin, who built the engine.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#28

Post by embedding-shape »

springtimesun wrote: post #21
pavlov wrote: post #10

Earlier quoted context omitted.

This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.

But, Kimi thought for 36k tokens before writing it.

And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#30

Post by Argonautlabs »

bluechair wrote: post #22

I missed the explanation for how the SSDs are connected. Maybe a dumb question.

SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here

https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...

Post reply on HN