Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

twitter.com

Post reply on HN
Print view View original

171–180 of 194 posts

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#171

Post by apitman »

weitendorf wrote: post #136
johnb231 wrote: post #108

Earlier quoted context omitted.

These are all "too big to host at home". I don't think that is the issue here. https://github.com/MoonshotAI/Kimi-K2/blob/main/docs/deploy_... "The smallest deployment unit for Kimi-K2 FP8 weights with 128k seqlen on mainstream H200 or H20 platform is a cluster with 16 GPUs with either Tensor Parallel (TP) or "data parallel + expert parallel" (DP+EP)." 16 GPUs costing ~$30k each. No one is running a ~$500k server at…

For most people, before it makes sense to just buy all the hardware yourself, you probably should be renting GPUs by the hour from the various providers serving that need. On Modal, I think should cost about $72/hr to serve Kimi K2 https://modal.com/pricing Once that's running it can serve the needs of many users/clients simultaneously. It'd be too expensive and underutilized for almost any individual to use regularl…

How does multi-user work, and how many users could it handle concurrently? My only experience is running much smaller models, and they easily peg my GPU at ~90 tokens/s. So maybe I could run 5-10 users at <10t/s? Does software like llama.cpp and ollama handle this?

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#172

Post by apitman »

MaxPock wrote: post #153
spaceman_2020 wrote: post #152

Earlier quoted context omitted.

The real users for these open source models are businesses that want something on premises for data privacy reasons Not sure if they’ll trust a Chinese model but dropping $50-100k for a quantized model that replaces, say, 10 paralegals is good enough for a law firm

An on-premise,open source Chinese model for my business,or a closed source American model from a company that's a defense contractor .Shouldn’t be too difficult a decision to make.

Even if they provide the code/data and not just the weights, aren't you taking their word for it that the weights were trained using that code, and not modified? Or is there some way to verify that?

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#173

Post by apitman »

jimjimwii wrote: post #167
refulgentis wrote: post #45

Earlier quoted context omitted.

You said "Good." then wrote a nice stirring bit about how having a bad experience with a 1T model will force people to try 4B/32B models. That seems separate from the post it was replying to, about 1T param models. If it is intended to be a reply, it hand waves about how having a bad experience with it will teach them to buy more expensive hardware. Is that "Good."? The post points out that if people are taught they…

Had you posted this comment in the early 90s about linux instead of local models, it would have made about the same amount of sense but aged just as poorly as this comment will. I'll remain here happily using 2.something tokens / second model.

But local aka desktop Linux is still an awful experience for most people. I use Arch btw

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#174

Post by apitman »

selfhoster11 wrote: post #170
CamperBob2 wrote: post #66

Earlier quoted context omitted.

That's pretty good. Are you running the real 600B+ parameter R1, or a distill, though?

The full thing, 671B. It loses some intelligence at 1.5 bit quantisation, but it's acceptable. I could actually go for around 3 bits if I max out my RAM, but I haven't done that yet.

I've seen people say the models get more erratic at higher (lower?) quantization levels. What's your experience been?

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#175

Post by apitman »

wiradikusuma wrote: post #15

I've only started using Claude, Gemini, etc in the last few months (I guess it comes with age, I'm no longer interested in trying the latest "tech"). I assume those are "non-agentic" models. From reading articles online, "agentic" means like you have a "virtual" Virtual Assistant with "hands" that can google, open apps, etc, on their own. Why not use existing "non-agentic" model and "orchestrate" them using LangChain…

I'm new too. Found this article helpful: https://crawshaw.io/blog/programming-with-agents

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#176

Post by the_precipitate »

ozgune wrote: post #57

This is a very impressive general purpose LLM (GPT 4o, DeepSeek-V3 family). It’s also open source. I think it hasn’t received much attention because the frontier shifted to reasoning and multi-modal AI models. In accuracy benchmarks, all the top models are reasoning ones: https://artificialanalysis.ai/ If someone took Kimi k2 and trained a reasoning model with it, I’d be curious how that model performs.

[dead]

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#177

Post by scottyeager »

c4pt0r wrote: post #5
simonw wrote: post #3

Big release - https://huggingface.co/moonshotai/Kimi-K2-Instruct model weights are 958.52 GB

Paired with programming tools like Claude Code, it could be a low-cost/open-source replacement for Sonnet

Here's a neat looking project that allows for using other models with Claude Code: https://github.com/musistudio/claude-code-router

I found that while looking for reports of the best agents to use with K2. The usual suspects like Cline and forks, Aider, and Zed should be interesting to test with K2 as well.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#178

Post by chithanh »

gpm wrote: post #93
summarity wrote: post #64

Earlier quoted context omitted.

Reasonable speeds are possible with 4bit quants on 2 512GB Mac Studios (MLX TB4 Ring - see https://x.com/awnihannun/status/1943723599971443134 ) or even a single socket Epyc system with >1TB of RAM (about the same real world memory throughput as the M Ultra). So $20k-ish to play with it. For real-world speeds though yeah, you'd need serious hardware. This is more of a "deploy your own stamp" model, less a "local" mod…

> or even a single socket Epyc system with >1TB of RAM How many tokens/second would this likely achieve?

KTransformers now supports Kimi K2 for MoE offloading

They claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory.

Re: Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model

#179

Post by MaxPock »

apitman wrote: post #172
MaxPock wrote: post #153

Earlier quoted context omitted.

An on-premise,open source Chinese model for my business,or a closed source American model from a company that's a defense contractor .Shouldn’t be too difficult a decision to make.

Even if they provide the code/data and not just the weights, aren't you taking their word for it that the weights were trained using that code, and not modified? Or is there some way to verify that?

I don't care .I'm hosting LLM and I can train or modify it the way I like. I'll have this authoritarian open source any day
Post reply on HN