Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

kimi.com

Post reply on HN
Print view View original

51–60 of 251 posts

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#51

Post by zozbot234 »

bertili wrote: post #48
Barathkanna wrote: post #46

A realistic setup for this would be a 16× H100 80GB with NVLink. That comfortably handles the active 32B experts plus KV cache without extreme quantization. Cost-wise we are looking at roughly $500k–$700k upfront or $40–60/hr on-demand, which makes it clear this model is aimed at serious infra teams, not casual single-GPU deployments. I’m curious how API providers will price tokens on top of that hardware reality.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

That's great for affordable local use but it'll be slow: even with the proper multi-node inference setup, the thunderbolt link will be a comparative bottleneck.

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#52

Post by embedding-shape »

bertili wrote: post #48
Barathkanna wrote: post #46

A realistic setup for this would be a 16× H100 80GB with NVLink. That comfortably handles the active 32B experts plus KV cache without extreme quantization. Cost-wise we are looking at roughly $500k–$700k upfront or $40–60/hr on-demand, which makes it clear this model is aimed at serious infra teams, not casual single-GPU deployments. I’m curious how API providers will price tokens on top of that hardware reality.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

I'd love to see the prompt processing speed difference between 16× H100 and 2× Mac Studio.

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#54

Post by Barathkanna »

bertili wrote: post #48
Barathkanna wrote: post #46

A realistic setup for this would be a 16× H100 80GB with NVLink. That comfortably handles the active 32B experts plus KV cache without extreme quantization. Cost-wise we are looking at roughly $500k–$700k upfront or $40–60/hr on-demand, which makes it clear this model is aimed at serious infra teams, not casual single-GPU deployments. I’m curious how API providers will price tokens on top of that hardware reality.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

That won’t realistically work for this model. Even with only ~32B active params, a 1T-scale MoE still needs the full expert set available for fast routing, which means hundreds of GB to TBs of weights resident. Mac Studios don’t share unified memory across machines, Thunderbolt isn’t remotely comparable to NVLink for expert exchange, and bandwidth becomes the bottleneck immediately. You could maybe load fragments experimentally, but inference would be impractically slow and brittle. It’s a very different class of workload than private coding models.

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#55

Post by Rebuff5007 »

pu_pe wrote: post #35

I don't get this "agent swarm" concept. You set up a task and they boot up 100 LLMs to try to do it in parallel, and then one "LLM judge" puts it all together? Is there anywhere I can read more about it?

I've also been quite skeptical, and I became even more skeptical after hearing a tech talk from a startup in this space [1].

I think the best way to think about it is that its an engineering hack to deal with a shortcoming of LLMs: for complex queries LLMs are unable to directly compute a SOLUTION given a PROMPT, but are instead able to break down the prompt to intermediate solutions and eventually solve the original prompt. These "orchestrator" / "swarm" agents add some formalism to this and allow you to distribute compute, and then also use specialized models for some of the sub problems.

[1] https://www.deepflow.com/

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#56

Post by tosh »

zmmmmm wrote: post #16

Curious what would be the most minimal reasonable hardware one would need to deploy this locally?

I think you can put a bunch of apple silicon macs with enough ram together

e.g. in an office or coworking space

800-1000 gb ram perhaps?

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#57

Post by zozbot234 »

Barathkanna wrote: post #54
bertili wrote: post #48

Earlier quoted context omitted.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

That won’t realistically work for this model. Even with only ~32B active params, a 1T-scale MoE still needs the full expert set available for fast routing, which means hundreds of GB to TBs of weights resident. Mac Studios don’t share unified memory across machines, Thunderbolt isn’t remotely comparable to NVLink for expert exchange, and bandwidth becomes the bottleneck immediately. You could maybe load fragments exp…

If "fast" routing is per-token, the experts can just reside on SSD's. the performance is good enough these days. You don't need to globally share unified memory across the nodes, you'd just run distributed inference.

Anyway, in the future your local model setups will just be downloading experts on the fly from experts-exchange. That site will become as important to AI as downloadmoreram.com.

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#58

Post by zozbot234 »

embedding-shape wrote: post #52
bertili wrote: post #48

Earlier quoted context omitted.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

I'd love to see the prompt processing speed difference between 16× H100 and 2× Mac Studio.

Prompt processing/prefill can even get some speedup from local NPU use most likely: when you're ultimately limited by thermal/power limit throttling, having more efficient compute available means more headroom.

Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

#60

Post by Barathkanna »

embedding-shape wrote: post #52
bertili wrote: post #48

Earlier quoted context omitted.

The other realistic setup is $20k, for a small company that needs a private AI for coding or other internal agentic use with two Mac Studios connected over thunderbolt 5 RMDA.

I'd love to see the prompt processing speed difference between 16× H100 and 2× Mac Studio.

I asked GPT for a rough estimate to benchmark prompt prefill on an 8,192 token input. • 16× H100: 8,192 / (20k to 80k tokens/sec) ≈ 0.10 to 0.41s • 2× Mac Studio (M3 Max): 8,192 / (150 to 700 tokens/sec) ≈ 12 to 55s

These are order-of-magnitude numbers, but the takeaway is that multi H100 boxes are plausibly ~100× faster than workstation Macs for this class of model, especially for long-context prefill.

Post reply on HN