Kimi Linear: An Expressive, Efficient Attention Architecture (2025)

Anthropic has more than $20 million in revenue so as per the k3 license they would need to enter a special commercial deal if they wanted to use it:

https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE

My expertise lies in deep learning theory, and yes, the "intelligence" is coming primarily from scaling up, among other things. There are good reasons for this, but essentially it comes down to taking advantage of a narrow statistical trick, where a very well-crafted model/optimizer pair that has a strong implicit bias toward simplicity can exhibit progressively increasing performance with respect to model size. Marcus Hutter's lab has shown that you can phrase this in terms of Solomonoff induction, so this bias is truly universally effective. An effective bias can continue to improve performance with larger model sizes by taking advantage of the curse of dimensionality in a way not dissimilar to how more data generally gives you a better answer (indeed, there is a duality taking place here, but I digress).

To be clear, it is an extremely narrow model class that can do this; we just got "lucky" and worked our way to it. That's why we still teach general statistical principles which often forbid this sort of behavior as a rule of thumb.

I'm out of the scene these days, and my field is more RL than LLMs, but my take is that all of this is telling us that meaning (and intelligence) is in the medium. If an LLM is a function mapping an input to some stateful internal representation that in-turn then gets mapped to an output, a smaller model probably doesn't have the capacity to learn that internal mapping, at least not from scratch. Or it would take a very long time, in same way that a naive sorting algorithm would still get there, but it might be computationally prohibitive in practice.

The bigger model has a better chance of gaining a foothold in that internal representation space where inputs are mapped to meanings and outputs, and eventually, it optimizes to the point where most of the weights aren't doing much. It's not immediately clear how much expressivity is required by the network to learn that space, but so far the answer seems to be in the billions of parameters.

The more interesting question to me is to what extent we should expect the models to be invariant to data. For instance, if I learn a certain type of analysis, that skill shouldn't depend on the data I'm looking at-- it should be repeatable for any given data of the same type/class. I'm curious to what extent skills are embedded in the weights versus data and "facts". My hunch for why mathematical reasoning and programming resulted in large step changes in model performance across the board is because these are inherently skills that are widely repeatable for a large class of tasks. The ability to express programmatic logic is invariant to both the language and the task at hand. And to me, that's how you get to smaller models: by focusing on the skills.

No claims here, though Algolia brought me here to alleged collateral damage: someone scraping HN a week back was unable to do their usual summary of a story and comments given a political refusal. https://news.ycombinator.com/item?id=48987120

This user tried Qwen (and DeepSeek, Kimi, GLM) ten days ago, called the response's language milquetoast: https://news.ycombinator.com/item?id=48964345

Comment from a user a month ago with their blog link on DeepSeek, which for comparison includes a generated poem about the evil killings of unarmed students by the USA at Kent State: https://news.ycombinator.com/item?id=48680612

(I vouch for none of these alleged tests.)

Of course when these tests used hosted models, different rules can apply. btw China is an amazing country with amazing people; my interest is in a little of everything, including free SotA-adjacent open software (thank you China!). Both respect their sovereignty to regulate software as used within their borders and hope labs there find value in serving Western users with the openness/transparency we (I) like to think we desire. I believe grappling with the most uncomfortable topics will be to their advantage in the long run. And ours (USA), too. The more we can divorce ourselves from bias, the more we can all win, I hope. My bias is towards humanity [being safe, happy, fulfilled...].

(While I might sound like someone who'd appreciate the model billed as "maximally truth seeking", unfortunately due to training or system prompting or something, Grok is trash unless you need to search Twitter or perhaps bypass botblocks. Or make "7K sex images of stepdaughter", I reference with apologies and deep sympathy to Jane Doe 4.)

It is _emerging_ in the sense that complex (sic non-linear) systems can exhibit unintended or unexpected behaviors.

It is _intelligent_ in the sense that it optimizes a thing that is hard for humans to not anthropomorphize.

Things like percolation theory, swarm theory, and others are similar topics in “complex systems”. Neural networks are interesting because they combine aspects of both complex systems and dynamical/adaptive systems (ie systems with a feedback loop).

A neural network at its very core is a function fitting algorithm. It stores matrices of parameters (ie weights and biases) such that every parameter (and combinations thereof) captures a relationship of your data in exactly the same way the slope and intercept are obtained through Linear Interpolation. All of the Regularization tricks applied just are attempts to incentivize a given parameter to not encode any trends too specifically (think an instance vs a type).

In this way you can ask yourself “how many aspects of your data are required to capture it adequately?” This is what scaling offers.