Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

arxiv.org

Post reply on HN
Print view View original

31–40 of 141 posts

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#31

Post by verdverm »

senko wrote: post #13
throwa356262 wrote: post #4

Earlier quoted context omitted.

For K3 read this instead: https://arxiv.org/abs/2607.24653 The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.

Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?

most new effort in training comes in the late phase with RL techniques

the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#33

Post by lacunary »

thomasahle wrote: post #24
pooyamo wrote: post #20

Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled? Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture? It's unintuitiv…

Here's one way it could happen: Let's say there's some circuit that does problem solving of the kind we call intelligence. We dont know what this circuit looks like, but it exists in our brain. Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit. But if you try to fit it with fewer parameters than it needs, you're just going to get some linear…

what is the approximation linear in?

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#34

Post by verdverm »

jlamberts wrote: post #27
pooyamo wrote: post #20

Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled? Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture? It's unintuitiv…

This is actually a well-known phenomenon in ML, called "The Bitter Lesson". > One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning. The full essay is worth a read, it's pre…

[flagged]

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#35

Post by iandanforth »

bratao wrote: post #16

I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791 ), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.

Is it just me or does this read like a re-implementation of LSTMs?

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#36

Post by hendiatris »

thomasahle wrote: post #24
pooyamo wrote: post #20

Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled? Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture? It's unintuitiv…

Here's one way it could happen: Let's say there's some circuit that does problem solving of the kind we call intelligence. We dont know what this circuit looks like, but it exists in our brain. Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit. But if you try to fit it with fewer parameters than it needs, you're just going to get some linear…

So basically a Nyquist rate type of concept.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#38

Post by lern_too_spel »

lacunary wrote: post #33
thomasahle wrote: post #24

Earlier quoted context omitted.

Here's one way it could happen: Let's say there's some circuit that does problem solving of the kind we call intelligence. We dont know what this circuit looks like, but it exists in our brain. Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit. But if you try to fit it with fewer parameters than it needs, you're just going to get some linear…

what is the approximation linear in?

In the output of this nonlinear model /s.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

#39

Post by serial_dev »

api wrote: post #28
WhitneyLand wrote: post #19

Earlier quoted context omitted.

False dichotomy right? Are Chinese labs impressively innovating? Clearly. However this doesn’t rule out possible gains due to distillation. I don’t know the degree of the latter but both things could certainly be true.

“Distillation” is just indirectly pirating the largely pirated training data used to train the original model. “You stole my warez!”

If we do it, it's training a model. When they do it, it's distillation attack. - Anthropic
Post reply on HN