Pretty cute pelican https://tools.simonwillison.net/svg-render#%3Csvg%20viewBox%...
Oops, here's a working link: https://gist.github.com/simonw/32a85e337fbc6ee935d10d89726c0...
Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
121–130 of 251 posts
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#122Earlier quoted context omitted.
The model absolutely can be run at home. There even is a big community around running large models locally: https://www.reddit.com/r/LocalLLaMA/ The cheapest way is to stream it from a fast SSD, but it will be quite slow (one token every few seconds). The next step up is an old server with lots of RAM and many memory channels with maybe a GPU thrown in for faster prompt processing (low two digits tokens/second). At t…
How do you split the model between multiple GPUs?
With "only" 32B active params, you don't necessarily need to. We're straying from common home users to serious enthusiasts and professionals but this seems like it would run ok on a workstation with a half terabyte of RAM and a single RTX6000.
But to answer your question directly, tensor parallelism. https://github.com/ggml-org/llama.cpp/discussions/8735 https://docs.vllm.ai/en/latest/configuration/conserving_memo...
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#123Have you all noted that the latest releases (Qwen3 max thinking, now Kimi k2.5) from Chinese companies are benching against Claude opus now and not Sonnet? They are truly catching up, almost at the same pace?
https://clocks.brianmoore.com
K2 is one of the only models to nail the clock face test as well. It’s a great model.
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#124Earlier quoted context omitted.
An LLM model only outputs tokens, so this could be seen as an extension of tool calling where it has trained on the knowledge and use-cases for "tool-calling" itself as a sub-agent.
Ok, so agent swarm = tool calling where the tool is a LLM call and the argument is the prompt
Sort of. It’s not necessarily a single call. In the general case it would be spinning up a long-running agent with various kinds of configuration — prompts, but also coding environment and which tools are available to it — like subagents in Claude Code.
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#125The "Deepseek moment" is just one year ago today! Coincidence or not, let's just marvel for a second over this amount of magic/technology that's being given away for free... and how liberating and different this is than OpenAI and others that were closed to "protect us all".
I mean, there are credible safety issues here. A Kimi fine-tune will absolutely be able to help people do cybersecurity related attacks - very good ones.
In a few years, or less, biological attacks and other sorts of attacks will be plausible with the help of these agents.
Chinese companies aren't humanitarian endeavors.
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#126The "Deepseek moment" is just one year ago today! Coincidence or not, let's just marvel for a second over this amount of magic/technology that's being given away for free... and how liberating and different this is than OpenAI and others that were closed to "protect us all".
I mean, there are credible safety issues here. A Kimi fine-tune will absolutely be able to help people do cybersecurity related attacks - very good ones. In a few years, or less, biological attacks and other sorts of attacks will be plausible with the help of these agents. Chinese companies aren't humanitarian endeavors.
[dead]
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#127Have you all noted that the latest releases (Qwen3 max thinking, now Kimi k2.5) from Chinese companies are benching against Claude opus now and not Sonnet? They are truly catching up, almost at the same pace?
https://clocks.brianmoore.com K2 is one of the only models to nail the clock face test as well. It’s a great model.
Cool comparison, but none of them get both the face and the time correct when I look at it.
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#128Earlier quoted context omitted.
> The model absolutely can be run at home. There even is a big community around running large models locally IMO 1tln parameters and 32bln active seems like a different scale to what most are talking about when they say localLLMs IMO. Totally agree there will be people messing with this, but the real value in localLLMs is that you can actually use them and get value from them with standard consumer hardware. I don't…
32B active is nothing special, there's local setups that will easily support that. 1T total parameters ultimately requires keeping the bulk of them on SSD. This need not be an issue if there's enough locality in expert choice for any given workload; the "hot" experts will simply be cached in available spare RAM.
When I've measured this myself, I've never seen a medium-to-long task horizon that would have expert locality such that you wouldn't be hitting the SSD constantly to swap layers (not to say it doesn't exist, just that in the literature and in my own empirics, it doesn't seem to be observed in a way you could rely on it for cache performance).
Over any task that has enough prefill input diversity and a decode phase thats more than a few tokens, its at least intuitive that experts activate nearly uniformly in the aggregate, since they're activated per token. This is why when you do something more than bs=1, you see forward passes light up the whole network.
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#129Earlier quoted context omitted.
One. Trillion. Even on native int4 that’s… half a terabyte of vram?! Technical awe at this marvel aside that cracks the 50th percentile of HLE, the snarky part of me says there’s only half the danger in giving something away nobody can run at home anyway…
The model absolutely can be run at home. There even is a big community around running large models locally: https://www.reddit.com/r/LocalLLaMA/ The cheapest way is to stream it from a fast SSD, but it will be quite slow (one token every few seconds). The next step up is an old server with lots of RAM and many memory channels with maybe a GPU thrown in for faster prompt processing (low two digits tokens/second). At t…
Its often pointed out in the first sentence of a comment how a model can be run at home, then (maybe) towards the end of the comment it’s mentioned how it’s quantized.
Back when 4k movies needed expensive hardware, no one was saying they could play 4k on a home system, then later mentioning they actually scaled down the resolution to make it possible.
The degree of quality loss is not often characterized. Which makes sense because it’s not easy to fully quantify quality loss with a few simple benchmarks.
By the time it’s quantized to 4 bits, 2 bits or whatever, does anyone really have an idea of how much they’ve gained vs just running a model that is sized more appropriately for their hardware, but not lobotomized?
Re: Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
#130Earlier quoted context omitted.
One. Trillion. Even on native int4 that’s… half a terabyte of vram?! Technical awe at this marvel aside that cracks the 50th percentile of HLE, the snarky part of me says there’s only half the danger in giving something away nobody can run at home anyway…
The model absolutely can be run at home. There even is a big community around running large models locally: https://www.reddit.com/r/LocalLLaMA/ The cheapest way is to stream it from a fast SSD, but it will be quite slow (one token every few seconds). The next step up is an old server with lots of RAM and many memory channels with maybe a GPU thrown in for faster prompt processing (low two digits tokens/second). At t…
>The model absolutely can be run at home.
There is a huge difference between "look I got it to answer the prompt: '1+1='"
and actually using it for anything of value.
I remember early on people bought Macs (or some marketing team was shoveling it), and proposing people could reasonably run the 70B+ models on it.
They were talking about 'look it gave an answer', not 'look this is useful'.
While it was a bit obvious that 'integrated GPU' is not Nvidia VRAM, we did have 1 mac laptop at work that validated this.
Its cool these models are out in the open, but its going to be a decade before people are running them at a useful level locally.