Kimi K3 Architecture Overview and Notes
sebastianraschka.com
Kimi K3 Architecture Overview and Notes
1–10 of 125 posts
Re: Kimi K3 Architecture Overview and Notes
#2Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
Re: Kimi K3 Architecture Overview and Notes
#3Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
Re: Kimi K3 Architecture Overview and Notes
#4Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.
> Seems to benchmark
yes, but in human usage the differences show up
Re: Kimi K3 Architecture Overview and Notes
#5Earlier quoted context omitted.
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews. > Seems to benchmark yes, but in human usage the differences show up
Would you happen to have a link to that interview? Sounds like an interesting read.
Re: Kimi K3 Architecture Overview and Notes
#6Earlier quoted context omitted.
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews. > Seems to benchmark yes, but in human usage the differences show up
Would you happen to have a link to that interview? Sounds like an interesting read.
It was posted to HN a few days back
1. https://news.ycombinator.com/item?id=49019012 (original chinese)
2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])
Re: Kimi K3 Architecture Overview and Notes
#7Earlier quoted context omitted.
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews. > Seems to benchmark yes, but in human usage the differences show up
Would you happen to have a link to that interview? Sounds like an interesting read.
It's related to this: https://news.ycombinator.com/item?id=49052912
Re: Kimi K3 Architecture Overview and Notes
#8Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
Re: Kimi K3 Architecture Overview and Notes
#9Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
It has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.
Re: Kimi K3 Architecture Overview and Notes
#10Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
This comment smells LLM generated.