Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

moonshotai.github.io

Post reply on HN
Print view View original

391–400 of 442 posts

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#391

Post by Der_Einzige »

shawndrost wrote: post #388
simonw wrote: post #309

Earlier quoted context omitted.

If you have 1,000 researchers working for your company and you constantly have dozens of different training runs in the go, overlapping each other, how would you split those salaries between those different runs? Calculating the cost in terms of GPU-hours is a whole lot easier from an accounting perspective. The papers I've seen that talk about training cost all do it in terms of GPU hours. The gpt-oss model card sai…

Do they include the costs of dead-end runs?

No, they don't! That's why the "5.5 million" deepseek V3 number as read by American investors was total bullshit (because investors ignored their astrik saying "only final training run")

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#392

Post by conradev »

embedding-shape wrote: post #377
yanhangyhy wrote: post #372

Earlier quoted context omitted.

I remember this thing. The tech is from America actually, decades ago. (Thorium). But they give up and china counties the work recent years

> The tech is from America actually, decades ago. (Thorium). I guess it depends on how you see it, but regardless, the people putting it to use today doesn't seem to be in the US. FWIW: > Thorium was discovered in 1828 by the Swedish chemist Jöns Jacob Berzelius during his analysis of a new mineral [...] In 1824, after more deposits of the same mineral in Vest-Agder, Norway, were discovered [...] While thorium was di…

The tech we’re talking about is using Thorium for useful work:

https://en.wikipedia.org/wiki/Thorium-based_nuclear_power#Hi...

https://en.wikipedia.org/wiki/Molten-Salt_Reactor_Experiment

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#393

Post by a-dub »

HarHarVeryFunny wrote: post #380
a-dub wrote: post #359

Earlier quoted context omitted.

no. truly open source models are wonderful and remarkable things that truly move the needle in education, understanding, distributed collaboration and the advancement of the state of the art. redefinition of the terminology reduces incentive to strive for the wonderful goal that they represent.

There is a big difference between open source for something like the linux kernel or gcc where anyone with a home PC can build it, and any non-trivial LLM where it takes cloud compute and costs a lot to train it. No hobbyist or educational institution is going to be paying for million dollar training runs, probably not even thousand dollar ones.

"too big to share." nope. sharing the finished soup base, even if well suited for inclusion in other recipes, is still different from sharing the complete recipe. sharing the complete recipe encourages innovation in soup bases, including bringing the cost down for making them from scratch.

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#394

Post by nickthegreek »

poszlem wrote: post #102
Philpax wrote: post #92

Earlier quoted context omitted.

yes yes Chinese models have Chinese censorship, we don't need to belabour this point every time

No, we need to belabour it every time.

100% agree with you. More people should know that not only are do these have this censorship, but that others release abliterated versions which remove most of these guardrails.

https://huggingface.co/blog/mlabonne/abliteration

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#395

Post by simonw »

Der_Einzige wrote: post #391
shawndrost wrote: post #388

Earlier quoted context omitted.

Do they include the costs of dead-end runs?

No, they don't! That's why the "5.5 million" deepseek V3 number as read by American investors was total bullshit (because investors ignored their astrik saying "only final training run")

Yeah, that's one of the most frustrating things about these published numbers. Nobody ever wants to share how much money they spent on runs that didn't produce a useful model.

As with staffing costs though it's hard to account for these against individual models. If Anthropic run a bunch of training experiments that help them discover a new training optimization, then use that optimization as part of the runs for the next Opus and Sonnet and Haiku (and every subsequent model for the lifetime of the company) how should the cost of that experimental run be divvied up?

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#396

Post by danielmarkbruce »

a-dub wrote: post #393
HarHarVeryFunny wrote: post #380

Earlier quoted context omitted.

There is a big difference between open source for something like the linux kernel or gcc where anyone with a home PC can build it, and any non-trivial LLM where it takes cloud compute and costs a lot to train it. No hobbyist or educational institution is going to be paying for million dollar training runs, probably not even thousand dollar ones.

"too big to share." nope. sharing the finished soup base, even if well suited for inclusion in other recipes, is still different from sharing the complete recipe. sharing the complete recipe encourages innovation in soup bases, including bringing the cost down for making them from scratch.

There is an enormous amount of information in the public domain about building models. In fact, once you get into the weeds you'll realize there is too much and in many cases (not all, but many) the very specific way something was done or what framework they used or what hardware configuration they had was just a function of what they have or have experience with etc. One could spend a lifetime just trying to repro olmo's work or a lot of the huggingface stuff....

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#397

Post by HarHarVeryFunny »

oofbey wrote: post #382
HarHarVeryFunny wrote: post #334

Earlier quoted context omitted.

It's not clear if the ultimate SLMs will come from teams with less computing resources directly building them, or from teams with more resources performing ablation studies etc on larger models to see what can be removed. I wouldn't care to guess what the limit is, but Karpathy was suggesting in his Dwarkesh interview that maybe AGI could be a 1B parameter model if reasoning is separated (to extent possible) from kno…

That’s backwards. New research and ideas are proven on small models. Lots and lots of ideas are tested that way. Good ideas get scaled up to show they still work on medium sized models. The very best ideas make their way into the code for the next huge training runs, which can cost tens or hundreds of millions of dollars. Not to nitpick words, but ablation is the practice of stripping out features of an algorithm or…

> Not to nitpick words, but ablation is the practice of stripping out features of an algorithm ...

Ablation generally refers to removing parts of a system to see how it performs without them. In the context of an LLM it can refer to training data as well as the model itself. I'm not saying it'd be the most cost-effective method, but one could certainly try to create a small coding model by starting with a large one that performs well, and seeing what can be stripped out of the training data (obviously a lot!) without impacting the performance.

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#398

Post by ares623 »

tonyhart7 wrote: post #333
ares623 wrote: post #70

Earlier quoted context omitted.

I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it.

"I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it." seems like you are here that not understand this Company want to replace human and won't need to pay massive salary

I understand the companies wanting it. I hate it, but I understand.

I don’t understand the humans wanting to be replaced though.

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#399

Post by riku_iki »

simonw wrote: post #304
riku_iki wrote: post #289

Earlier quoted context omitted.

You either don't know which training data was used for say chatgpt oss, or training data can be included into some open dataset like pile or similar. I think this test is very unreliable, and even if someone come to such conclusion, not clear what is the value of such conclusion, and if that someone can be trusted.

My intuition tells me it is vanishingly unlikely that any of the major AI labs - including the Chinese ones - have fine-tuned someone else's model and claimed that they trained it from scratch and got away with it. Maybe I'm wrong about that, but I've never heard any of the AI training experts (and they're a talkative bunch) raise that as a suspicion. There have been allegations of distillation - where models are par…

Additionally, it would be interesting to know if there is dynamics in opposite directions, US corps (oai, xai) can now incorporate Chinese models into their core models as one/several expert towers.

Re: Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

#400

Post by riku_iki »

lxe wrote: post #176
vlovich123 wrote: post #78

Earlier quoted context omitted.

Or this problem made it into the training data set.

It's an old problem, and it, along with many of the answers are in many recent data sets.

I assume training set components have also priorities, low priority data goes to training very few times at the beginning of pretraining, while higher priority data is trained on multiple times until the end.
Post reply on HN