Earlier quoted context omitted.
And Europeans don't it because quite frankly, we're not really doing anything particularly impressive with AI sadly.
[flagged]
What a load of tripe.
Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
151–160 of 442 posts
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#152Four independent Chinese companies released extremely good open source models in the past few months (DeepSeek, Qwen/Alibaba, Kimi/Moonshot, GLM/Z.ai). No American or European companies are doing that, including titans like Meta. What gives?
I get what you mean, but OpenAI did release the gpt-oss in August, just three months ago. I've had a very good experience with those models.
https://openai.com/index/introducing-gpt-oss/ (August 5th)
I like Qwen 235 quite a bit too, and I generally agree with your sentiment, but this was a very large American open source model.
Unless we're getting into the complications on what "open source" model actually means, in which case I have no clue if these are just open weight or what.
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#153Earlier quoted context omitted.
Sure, but that's the point ... today's locally runnable models are a long way behind SOTA capability, so it'd be nice to see more research and experimentation in that direction. Maybe a zoo of highly specialized small models + agents for S/W development - one for planning, one for coding, etc?
If I understand transformers properly, this is unlikely to work. The whole point of “Large” Language Models is that you primarily make them better by making them larger, and when you do so, they get better at both general and specific tasks (so there isn’t a way to sacrifice generality but keep specific skills when training a small models). I know a lot of people want this (Apple really really wants this and is pouri…
Actually there are ways you might get on device models to perform well. It is all about finding ways to have a smaller number of weights work efficiently.
One way is reusing weights in multiple decoders layers. This works and is used in many on-device models.
It is likely that we can get pretty high performance with this method. You can also combine this with low parameter ways to create overlapped behavior on the same weights as well, people had done LORA on top of shared weights.
Personally I think there are a lot of potential ways that you can cause the same weights to exhibit "overloaded" behaviour in multiple places in the same decoder stack.
Edit: I believe this method is used a bit for models targeted for the phone. I don't think we have seen significant work on people targeting say a 3090/4090 or similar inference compute size.
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#154Earlier quoted context omitted.
If I understand transformers properly, this is unlikely to work. The whole point of “Large” Language Models is that you primarily make them better by making them larger, and when you do so, they get better at both general and specific tasks (so there isn’t a way to sacrifice generality but keep specific skills when training a small models). I know a lot of people want this (Apple really really wants this and is pouri…
Yeah - the whole business model of companies like OpenAI and Anthropic, at least at the moment, seems to be that the models are so big that you need to run them in the cloud with metered access. Maybe that could change in the future to sale or annual licence business model if running locally became possible. I think scale helps for general tasks where the breadth of capability may be needed, but it's not so clear tha…
No I don’t think it’s a business model thing, I’m saying it may be a technical limitation of LLMs themselves. Like, that that there’s no way to “order a la carte” from the training process, you either get the buffet or nothing, no matter how hungry you feel.
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#155Earlier quoted context omitted.
The Chinese are doing it because they don't have access to enough of the latest GPUs to run their own models. Americans aren't doing this because they need to recoup the cost of their massive GPU investments.
And Europeans don't it because quite frankly, we're not really doing anything particularly impressive with AI sadly.
actually Mistral is pretty good and catching up as the other leading models stagnate - the coding and OCR is particularly good
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#156Earlier quoted context omitted.
I know first hand companies paying them. Chinese internal software market is gigantic. Full of companies and startups that have barely made into a single publication in the west.
Of course they are paying them. That’s not my point. My point is this is the only way for them to gain market share and they need Western users to train future models. They have to give them away. I’d be shocked if compute costs are not heavily subsidized by CCP.
But the CCP only has access to the US market because they joined the WTO, but when they joined the WTO they signed a treaty that they wouldn't do things like that.
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#157Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#158How does one effectively use something like this locally with consumer-grade hardware?
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#159It's good to see more competition, and open source, but I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a much smaller LLM + agent as opposed to a trillion parameter one. The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. The original mission OpenAI had, since abandoned, was to have AI benefit all of humanity, and…
The electricity cost to run these models locally is already more than equivalent API cost.
That's going to depend on how small the model can be made, and how much you are using it.
If we assume that running locally meant running on a 500W consumer GPU, then the electricity cost to run this non-stop 8 hours a day for 20 days a month (i.e. "business hours") would be around $10-20.
This is about the same as OpenAI or Anthropics $20/mo plans, but for all day coding you would want their $100 or $200/mo plans, and even these will throttle you and/or require you to switch to metered pricing when you hit plan limits.
Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
#160Earlier quoted context omitted.
If you want to do it at home, ik_llama.cpp has some performance optimizations that make it semi-practical to run a model of this size on a server with lots of memory bandwidth and a GPU or two for offload. You can get 6-10 tok/s with modest hardware workstation hardware. Thinking chews up a lot of tokens though, so it will be a slog.
What kind of server have you used to run a trillion parameter model? I'd love to dig more into this.
Hi Simon. I have a Xeon W5-3435X with a 768GB of DDR5 across 8 channels, iirc it's running at 5800MT/s. It also has 7x A4000s, water cooled to pack them into a desktop case. Very much a compromise build, and I wouldn't recommend Xeon sapphire rapids because the memory bandwidth you get in practice is less than half of what you'd calculate from the specs. If I did it again, I'd build an EPYC machine with 12 channels of DDR5 and put in a single rtx 6000 pro blackwell. That'd be a lot easier and probably a lot faster.
There's a really good thread on level1techs about running DeepSeek at home, and everything there more-or-less applies to Kimi K2.
https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-hom...