Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

71–80 of 151 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#71

Post by Capricorn2481 »

redox99 wrote: post #70
Capricorn2481 wrote: post #66

Earlier quoted context omitted.

I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now. You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s

Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.

Maybe. Do you have an example of a prompt you've done recently?

I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#72

Post by NooneAtAll3 »

Argonautlabs wrote: post #54
walrus01 wrote: post #50

Now imagine the token/s rate decline after context fill at 200,000+ context.

Fair, and we didn't measure it. Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand. The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass, so 200k tokens of pr…

what's the main limitation on context size?

4.4k seems... I just realized I have no sense of scale whatsoever

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#73

Post by jgalt212 »

RugnirViking wrote: post #64
jgalt212 wrote: post #33

Earlier quoted context omitted.

A medium prompt = 1 million tokens?

How many times have you had a model start compacting already before getting back to you? Most have 1 million context window. It's happened to me occasionally

I don't use agents. I like to code and AI with a REPL.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#74

Post by Argonautlabs »

BoingBoomTschak wrote: post #68

Even if that's impressive, the README is low SNR slop as usual...

Thank you.

Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#75

Post by anigbrowl »

Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

So we meet again, wall of text. I thought I had killed you in Istanbul.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#76

Post by NooneAtAll3 »

Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

if anyone else is lost at how big/small 512 tokens are - this wall of text is 843 tokens, according to https://token-calculator.net

so... enough for 1 question + 1 long answer?

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#77

Post by Argonautlabs »

NooneAtAll3 wrote: post #72
Argonautlabs wrote: post #54

Earlier quoted context omitted.

Fair, and we didn't measure it. Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand. The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass, so 200k tokens of pr…

what's the main limitation on context size? 4.4k seems... I just realized I have no sense of scale whatsoever

Memory, not the model.

The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs

(we found the hard way that squeezing those makes the verifier reject wide batches and decode falls to single-token steps).

With the current reserves the engine admits ~4.4k tokens; that's a configuration ceiling you can raise by giving the cache more of the 128 GB and accepting less headroom elsewhere.

K3 itself supports far longer contexts — but see the prefill caveat above: on this setup long prompts cost minutes per 512 tokens until the scheduling fix lands.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#78

Post by coldtea »

Capricorn2481 wrote: post #61
redox99 wrote: post #42

Earlier quoted context omitted.

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

> At 1t/s it's still faster than humans for a lot of tasks Which tasks? I think you're underestimating how token hungry current proposed workflows are.

Doesn't matter which task. Compare it with about 40-50t/s an LLM oneshots with, and it, and whatever task now takes X time, takes X * 40-50 with this.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#79

Post by NooneAtAll3 »

sixothree wrote: post #36
roadside_picnic wrote: post #13

Earlier quoted context omitted.

I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top. The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an i…

Gen X here. Having grown up in the 80's I remember multiple occasions where someone would ask "what are you going to do with a computer?". As in, what could a computer possibly be useful for. Just imagine someone asking this question today. It would probably be more shaped like the comment you are responding to. What are you going to do with a computer? I've always hated this attitude. We do these things because they…

> "what are you going to do with a computer?"

> Just imagine someone asking this question today.

to be fair, considering how young folks live whole life in a smartphone nowadays...

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#80

Post by redox99 »

Capricorn2481 wrote: post #71
redox99 wrote: post #70

Earlier quoted context omitted.

Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.

Maybe. Do you have an example of a prompt you've done recently? I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.

Stuff along the lines of implement controller service and tests for the following endpoints:

- list of many endpoints with the JSON they receive and return and description of what they need to achieve

Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.

Post reply on HN