Local AI is going to win. Not because it's cheaper btw.
Running Kimi K3 on a M1 Max
71–80 of 97 posts
Re: Running Kimi K3 on a M1 Max
#72Earlier quoted context omitted.
0.3tx per second is decent speed? And it gets worse with every token.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, ar…
Would it even finish over a few weeks?
For AI agents this would take a year
Re: Running Kimi K3 on a M1 Max
#73Earlier quoted context omitted.
0.3tx per second is decent speed? And it gets worse with every token.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, ar…
No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
Re: Running Kimi K3 on a M1 Max
#74Earlier quoted context omitted.
I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface.
I had my clanker implement this idea in a standalone Rust server that speaks IMAP and SMTP and proxies your emails to an OpenAI endpoint you configure: https://tangled.org/clee.sh/posthorn Works in mutt; other MUAs may vary.
lovely! I'm a fan of asynchronous LLM systems like this. There are times where I prefer slow, complete responses to immediate replies.
For example there's a billboard on 101 for Poppy Bank offering 4% interest and I asked my daily LLM to look into it through a voice note. The next morning, lo and behold it says it's an advertised rate, hard to actually get, and businesses aren't eligible. OK, done. Better than getting the response while I'm driving. And even if I wasn't driving, there's a level of, how do you say, it's easy to drop it the next day when I see it vs getting engrossed in the research.
Re: Running Kimi K3 on a M1 Max
#75Earlier quoted context omitted.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, ar…
No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
The OpenAI and Anthropic batch APIs give you a 50% discount.
This is like 98%.
Re: Running Kimi K3 on a M1 Max
#76Earlier quoted context omitted.
0.3tx per second is decent speed? And it gets worse with every token.
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, ar…
I read about its format being tough to compress further. What should we expect in terms of creating smaller models to run locally?
Re: Running Kimi K3 on a M1 Max
#77Re: Running Kimi K3 on a M1 Max
#78Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop. Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at leas…
0.01 tk/s on an M1 Max is not "nearly". This is completely unusable, and in no way cost effective. 0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS. [1] https://www.kimi.com/resources/kimi-k3-pricing
Ok in terms of running a 2.8T parameter model, that's true.
Looking more broadly though, a model I can run on my laptop (Gemma 4) is ~4 points away from GPT-5.3 codex or Sonnet 4.5 on arena.ai LLM leaderboard. Those models were SOTA less than a year ago.
Re: Running Kimi K3 on a M1 Max
#79Earlier quoted context omitted.
No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
The OpenAI and Anthropic batch APIs give you a 50% discount. This is like 98%.
I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
Re: Running Kimi K3 on a M1 Max
#80Going to try this on my M1 Ultra 128gb.
The point of these engineering tricks is to see the envelope of what's possible. You can use these tricks to both run a bigger model on smaller hardware or run a smaller model on smaller hardware.