Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

41–50 of 157 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#41

Post by IgorPartola »

schmorptron wrote: post #38
lukeduff wrote: post #19

Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy

It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably

My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#42

Post by redox99 »

schmorptron wrote: post #38
lukeduff wrote: post #19

Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy

It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#43

Post by pianopatrick »

voidnullvalue wrote: post #3

But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly

I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no""

Doing something like that at 1 token per second may be useful.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#44

Post by batperson »

schmorptron wrote: post #38
lukeduff wrote: post #19

Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy

It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably

The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).

GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#49

Post by Argonautlabs »

saejox wrote: post #32

make it 4x40 raid-0 ssds to achieve 40 tps. or 40 macbooks with each 4 ssd. to get 40 tps.

Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum.

Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with

Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.

What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.

Numbers in results/SCALING.md and results/PREFILL.md.

Post reply on HN