Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi Linear: An Expressive, Efficient Attention Architecture

github.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#21

Post by samus »

muragekibicho wrote: post #8
textembedding wrote: post #5

125 upvotes with 2 comments is kinda sus

Lots of model releases are like this. We can only upvote. We can't run the model on our personal computers. We can neither test their 'Efficient Attention' concept on our personal computers. Honestly, it would take 24 hours just to download the 98 GB model if I wanted to try it out (assuming I had a card with 98 GB of ram).

We very much can, especially such a Mixture of Experts model with only 3B activated parameters.

With an RTX 3070 (7GB GRAB VRAM), 32 GB RAM and an SSD I can run such models at speeds tolerable for casual use.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#22

Post by samus »

uniqueuid wrote: post #11
amoskvin wrote: post #7

any hardware recommendations? how much memory do we need to this?

You will effectively want a 48GB card or more for quantized versions, otherwise you won't have meaningful space left for the KV cache. Blackwell and above is generally a good idea to get faster hardware support for 4b (some recent models took some time to ship for older architectures, gpt-oss IIRC).

This is a Mixture of Experts model with only 3B activated parameters. But I agree that for the intended usage scenario VRAM for the KV cache is the real limitation.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#23

Post by embedding-shape »

hirako2000 wrote: post #18
logicartisan wrote: post #17

Amazing how fast AI keeps improving, every new model feels like a big step forward

It solely is improving on efficiency. While it is extremely valuable given the disproportionate (to value) costs of these things, your statement almost sounds like it has improved an even more challenging aspect, pushing performance.

It's a generic comment that I don't think is even specifically about Kimi Linear or this submission, you could leave the same comment on almost any AI/ML submission and it'd say the same amount and be as relevant/irrelevant.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#24

Post by giancarlostoro »

hirako2000 wrote: post #18
logicartisan wrote: post #17

Amazing how fast AI keeps improving, every new model feels like a big step forward

It solely is improving on efficiency. While it is extremely valuable given the disproportionate (to value) costs of these things, your statement almost sounds like it has improved an even more challenging aspect, pushing performance.

I've uh said this a few times. But AI is a bunch of people overpaying CS students to implement old algorithms and then realizing that they need Software Engineers to optimize the existing known systems. Most of AI (if not ALL of it) as we know it today has been coded for decades, we just never had the hardware for it.

A lot of the optimizations are not some ground breaking new way to program, they're known techniques to any Software Engineer or Systems Engineer.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#25

Post by embedding-shape »

giancarlostoro wrote: post #24
hirako2000 wrote: post #18

Earlier quoted context omitted.

It solely is improving on efficiency. While it is extremely valuable given the disproportionate (to value) costs of these things, your statement almost sounds like it has improved an even more challenging aspect, pushing performance.

I've uh said this a few times. But AI is a bunch of people overpaying CS students to implement old algorithms and then realizing that they need Software Engineers to optimize the existing known systems. Most of AI (if not ALL of it) as we know it today has been coded for decades, we just never had the hardware for it. A lot of the optimizations are not some ground breaking new way to program, they're known techniques…

> A lot of the optimizations are not some ground breaking new way to program

Hindsight is a bitch huh? Everything looks simple now once people proved it kind of works, but I think you over-simplify "how easy it is".

Lots of stuff in ML, particularly recent ~5 years or so, haven't been "implementing old algorithms" although of course everything is based on the research that happened in the past, we're standing on the shoulders of giants and all that.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#27

Post by embedding-shape »

samus wrote: post #21
muragekibicho wrote: post #8

Earlier quoted context omitted.

Lots of model releases are like this. We can only upvote. We can't run the model on our personal computers. We can neither test their 'Efficient Attention' concept on our personal computers. Honestly, it would take 24 hours just to download the 98 GB model if I wanted to try it out (assuming I had a card with 98 GB of ram).

We very much can, especially such a Mixture of Experts model with only 3B activated parameters. With an RTX 3070 (7GB GRAB VRAM), 32 GB RAM and an SSD I can run such models at speeds tolerable for casual use.

How many tok/s are you getting (with any runtime) with either the Kimi-Linear-Instruct or Kimi-Linear-Base on your RTX 3070?

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#28

Post by embedding-shape »

Der_Einzige wrote: post #20
muragekibicho wrote: post #8

Earlier quoted context omitted.

Lots of model releases are like this. We can only upvote. We can't run the model on our personal computers. We can neither test their 'Efficient Attention' concept on our personal computers. Honestly, it would take 24 hours just to download the 98 GB model if I wanted to try it out (assuming I had a card with 98 GB of ram).

People here absolutely can afford the ~2 dollars an hour of cloud rental costs for an H100 or even 8 (OCI has cheap H100 nodes). Most people are too lazy to even try and thank goodness for it because I prefer my very high salaries as someone who isn’t too lazy to spin up a cloud instance.

Not to mention some of us have enough disposable income to buy a RTX Pro 6000 so we can run our stuff locally and finally scale up our model training a little bit.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#29

Post by oxqbldpxo »

I switched from chatgpt to Perplexity; and now to Kimi K2, after reading an article here explaining that all the fear around some of the Chinese models spying and so on.. is simply not true. I have to say that in my experience Kimi K2 is way better than perplexity. I hope we can get our act together. Seems that building this Ai's requires a level of collaboration that is in opposition to greed.

Re: Kimi Linear: An Expressive, Efficient Attention Architecture

#30

Post by embedding-shape »

oxqbldpxo wrote: post #29

I switched from chatgpt to Perplexity; and now to Kimi K2, after reading an article here explaining that all the fear around some of the Chinese models spying and so on.. is simply not true. I have to say that in my experience Kimi K2 is way better than perplexity. I hope we can get our act together. Seems that building this Ai's requires a level of collaboration that is in opposition to greed.

> after reading an article here explaining that all the fear around some of the Chinese models spying and so on.. is simply not true

Not doubting they aren't spying on people, but regardless, how would you really know? Are you basing this on that no Chinese police has visited you, or how would you really know if it's "simply true" or not?

With that said, I use plenty of models coming out of China too with no fear, but I'm also using them locally, not cloud platforms.

Post reply on HN