Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
1–10 of 141 posts
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#2(2025)
As it's 9 months old and they just had a major model release
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#3(2025) As it's 9 months old and they just had a major model release
I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#4(2025) As it's 9 months old and they just had a major model release
For K3 read this instead: https://arxiv.org/abs/2607.24653
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#5(2025) As it's 9 months old and they just had a major model release
Rather under-discussed back then: https://news.ycombinator.com/item?id=45766937
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#6(2025) As it's 9 months old and they just had a major model release
For K3 read this instead: https://arxiv.org/abs/2607.24653 The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#7Earlier quoted context omitted.
For K3 read this instead: https://arxiv.org/abs/2607.24653 The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
Same with DeepSeek papers, they are a joy to read.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#8Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#9Earlier quoted context omitted.
I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.
It's not the same KDA as used in Kimi Linear, though.
What's the difference? They are both called Kimi Delta Attention.
Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
#10Another banger from Zhang et. al