Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

Post by handzhiev » Sat, Jul 12, 2025, 8:51 PM UTC

vessenes wrote: post #62

I tried Kimi on a few coding problems that Claude was spinning on. It’s good. It’s huge, way too big to be a “local” model — I think you need something like 16 H200s to run it - but it has a slightly different vibe than some of the other models. I liked it. It would definitely be useful in ensemble use cases at the very least.

Post by gpm » Sat, Jul 12, 2025, 9:10 PM UTC

summarity wrote: post #64

vessenes wrote: post #62

I tried Kimi on a few coding problems that Claude was spinning on. It’s good. It’s huge, way too big to be a “local” model — I think you need something like 16 H200s to run it - but it has a slightly different vibe than some of the other models. I liked it. It would definitely be useful in ensemble use cases at the very least.

Reasonable speeds are possible with 4bit quants on 2 512GB Mac Studios (MLX TB4 Ring - see https://x.com/awnihannun/status/1943723599971443134 ) or even a single socket Epyc system with >1TB of RAM (about the same real world memory throughput as the M Ultra). So $20k-ish to play with it. For real-world speeds though yeah, you'd need serious hardware. This is more of a "deploy your own stamp" model, less a "local" mod…

Post by teiferer » Sat, Jul 12, 2025, 9:17 PM UTC

echelon wrote: post #83

satvikpendem wrote: post #75

This is not open source, they have a "modified MIT license" where they have other restrictions on users over a certain threshold. Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue, you…

> This is not open source OSI purism is deleterious and has led to industry capture. Non-viral open source is simply a license for hyperscalers to take advantage. To co-opt offerings and make hundreds of millions without giving anything back. We need more "fair source" licensing to support sustainable engineering that rewards the small ICs rather than mega conglomerate corporations with multi-trillion dollar market c…

Post by fzzzy » Sat, Jul 12, 2025, 9:19 PM UTC

segmondy wrote: post #92

refulgentis wrote: post #84

Earlier quoted context omitted.

I write a local LLM client, but sometimes, I hate that local models have enough knobs to turn that people can advocate they're reasonable in any scenario - in yesterday's post re: Kimi k2, multiple people spoke up that you can "just" stream the active expert weights out of 64 GB of RAM, and use the lowest GGUF quant, and then you get something that rounds to 1 token/s, and that is reasonable for use. Good on you for…

I do deepseek at 5tk/sec at home and I'm happy with it. I don't need to do agent stuff to gain from it, I was saving to eventually build out enough to run it at 10tk/sec, but with kimi k2, plan has changed and the savings continue with a goal to run it at 5 tk/sec at home.

Post by satvikpendem » Sat, Jul 12, 2025, 9:28 PM UTC

echelon wrote: post #83

satvikpendem wrote: post #75

This is not open source, they have a "modified MIT license" where they have other restrictions on users over a certain threshold. Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue, you…

> This is not open source OSI purism is deleterious and has led to industry capture. Non-viral open source is simply a license for hyperscalers to take advantage. To co-opt offerings and make hundreds of millions without giving anything back. We need more "fair source" licensing to support sustainable engineering that rewards the small ICs rather than mega conglomerate corporations with multi-trillion dollar market c…