Kimi Linear: An Expressive, Efficient Attention Architecture

Lots of model releases are like this. We can only upvote. We can't run the model on our personal computers. We can neither test their 'Efficient Attention' concept on our personal computers.

Honestly, it would take 24 hours just to download the 98 GB model if I wanted to try it out (assuming I had a card with 98 GB of ram).

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#9

Post by quotemstr » Fri, Oct 31, 2025, 8:59 AM UTC

eXpl0it3r wrote: post #6

For the uninitiated, what's a "hybrid linear attention architecture"?

1/4 of their layers are conventional quadratic attention

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#10

Post by meowface » Fri, Oct 31, 2025, 9:32 AM UTC

quotemstr wrote: post #9

eXpl0it3r wrote: post #6

For the uninitiated, what's a "hybrid linear attention architecture"?

1/4 of their layers are conventional quadratic attention

Could someone explain every term in this subthread in a very simple way to someone who basically only knows "transformers are a neural network architecture that use something called 'attention' to consider the entire input the whole time or something like that", and who does not understand what "quadratic" even means in a time complexity or mathematical sense beyond that "quad" has something to do with the number four.

I am aware I could Google it all or ask an LLM, but I'm still interested in a good human explanation.