You currently can't run a 2.8T locally; there's just no way. So, it's a good start.
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
51–60 of 153 posts
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#52I wonder if faster SSDs would help? In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s.
What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#53But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly
I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no"" Doing something like that at 1 token per second may be useful.
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token,
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#54Now imagine the token/s rate decline after context fill at 200,000+ context.
Fair, and we didn't measure it.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#55Earlier quoted context omitted.
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#56Earlier quoted context omitted.
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.
[deleted]
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#57make it 4x40 raid-0 ssds to achieve 40 tps. or 40 macbooks with each 4 ssd. to get 40 tps.
yeah raid or not you still get the hard limitation by the pcie lanes
it is even worse with 40 macbooks.
if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#58Earlier quoted context omitted.
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently). GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a…
This is incredible! Are there other big players in this space (freezing models to silicon)?
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#59I wonder if faster SSDs would help? In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#60Earlier quoted context omitted.
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s).
I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something.
Its not a problem of the model of course, but it seems impractical atm.
I wonder if anyone else had this kind of thing happening.