tokens/second, no, more like seconds/token
The link beat you to it:
> It is not fast — about 16 seconds per token on our M1 Max
Running Kimi K3 on a M1 Max
61–70 of 97 posts
Re: Running Kimi K3 on a M1 Max
#62Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop. Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at leas…
0.01 tk/s on an M1 Max is not "nearly". This is completely unusable, and in no way cost effective.
0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS.
Re: Running Kimi K3 on a M1 Max
#63Re: Running Kimi K3 on a M1 Max
#640.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface.
I had my clanker implement this idea in a standalone Rust server that speaks IMAP and SMTP and proxies your emails to an OpenAI endpoint you configure: https://tangled.org/clee.sh/posthorn
Works in mutt; other MUAs may vary.
Re: Running Kimi K3 on a M1 Max
#65Earlier quoted context omitted.
'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?' - bunch of people only ~4 years ago
I questioned the speed not the output quality, that is another discussion
It applies to speed too. The project paves way for more optimization at many layers overtime.
Re: Running Kimi K3 on a M1 Max
#66Why not train another smaller LLM to give the same answers as Kimi K3?
why not zip the entire internet to 1MB so everyone can have a copy? because it is not possible - we don't know if it is possible. I mean we know it is impossible, but we don't know if it is possible to do it with acceptable quality loss.
Re: Running Kimi K3 on a M1 Max
#67SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.
0.3tx per second is decent speed? And it gets worse with every token.
That's not what he said. He said with 2x the hardware it will be faster.
Re: Running Kimi K3 on a M1 Max
#680.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
You can't improve what you can't measure.
Consider this like if it were the first test
Re: Running Kimi K3 on a M1 Max
#69SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.
0.3tx per second is decent speed? And it gets worse with every token.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them.
192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
Re: Running Kimi K3 on a M1 Max
#70how much money in hardware would it cost to get 100tps?