If you want to believe that the success of Kimi is about distillation attacks, ignore this.
Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
11–20 of 141 posts
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#12Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#13(2025) As it's 9 months old and they just had a major model release
For K3 read this instead: https://arxiv.org/abs/2607.24653 The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#14If you want to believe that the success of Kimi is about distillation attacks, ignore this.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#15If you want to believe that the success of Kimi is about distillation attacks, ignore this.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#16I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#17Earlier quoted context omitted.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen). Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Google is apparently taking a different stance and offering distillation as a paid product
https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#18Earlier quoted context omitted.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen). Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#19If you want to believe that the success of Kimi is about distillation attacks, ignore this.
False dichotomy right?
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#20Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.