Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 Architecture Overview and Notes

sebastianraschka.com

Post reply on HN
Print view View original

81–90 of 125 posts

Re: Kimi K3 Architecture Overview and Notes

#81

Post by desterothx »

spider-mario wrote: post #77
nl wrote: post #67

Earlier quoted context omitted.

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

Isn’t this a bit like saying “ ‘open object files are not open source’ is a complete misunderstanding” because “You can’t rebuild the executable from scratch, but you can’t do this even with the source code anyway (because of build nondeterminism / compiler versions / etc.)”?

not really, compiling isnt a heuristic problem, it has a lot less randomness involved

Re: Kimi K3 Architecture Overview and Notes

#82

Post by eru »

nl wrote: post #67
mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

> (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))

They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomness, like concurrent execution.)

Re: Kimi K3 Architecture Overview and Notes

#83

Post by marcyb5st »

calebkaiser wrote: post #45
mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented…

The exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.

Re: Kimi K3 Architecture Overview and Notes

#84

Post by eru »

desterothx wrote: post #81
spider-mario wrote: post #77

Earlier quoted context omitted.

Isn’t this a bit like saying “ ‘open object files are not open source’ is a complete misunderstanding” because “You can’t rebuild the executable from scratch, but you can’t do this even with the source code anyway (because of build nondeterminism / compiler versions / etc.)”?

not really, compiling isnt a heuristic problem, it has a lot less randomness involved

Depends on your compiler. You could have a compiler that deliberately uses randomised algorithms. They are often faster and easier to understand and write.

Though in practice you can get all the benefits of both determinism and (that kind of) randomisation by using a PRNG and saving the seed you are using.

It's an open question roughly on par with P vs NP whether true randomisation is ever necessary, or whether PRNGs are enough. So far we haven't found any problem or algorithm where true RNG is necessary and good PRNG ain't enough.

Re: Kimi K3 Architecture Overview and Notes

#85

Post by eru »

marcyb5st wrote: post #83
calebkaiser wrote: post #45

Earlier quoted context omitted.

Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented…

The exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.

> Should have minimal impact on the final outcome, [...]

I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.

Re: Kimi K3 Architecture Overview and Notes

#86

Post by boomskats »

anon373839 wrote: post #54
FinchNova12 wrote: post #49

Earlier quoted context omitted.

"Kimi is largely a byproduct of distillation" and "Kimi is introducing new and novel approaches" are not mutually exclusive, and I'm not sure it's clear from the paper how much of the improvement comes from the new approaches. So I wouldn't take the new approaches to be much evidence about whether the distillation attacks occurred.

> distillation attacks Like the distillation “attacks” Anthropic conducted against literally millions and millions of different creators?

Or against all those old books that they ingested and then literally destroyed

Re: Kimi K3 Architecture Overview and Notes

#88

Post by rekpero »

kimi K3 felt on par with my claude opus 4.8 in my first tests and is exceptionally good at frontend and website motion design, but I still find it hard to switch back and forth between claude and kimi, and i’m not sure if others are experiencing the same thing.

Re: Kimi K3 Architecture Overview and Notes

#89

Post by alightsoul »

eru wrote: post #82
nl wrote: post #67

Earlier quoted context omitted.

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

> (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!)) They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomn…

Do AI labs even keep track of seeds? They don't even keep the checkpoints most of the time.

Re: Kimi K3 Architecture Overview and Notes

#90

Post by embedding-shape »

nl wrote: post #67
mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

> You can't rebuild it from scratch

> It's entirely reproducible from the available documentation

You have a very interesting understanding of "reproducibility", I'll give you that :)

But even with that, there are plenty of technical details (especially in regards to the training process) missing from the tech report that leads to these weights not being reproducible in any sense of that word.

Post reply on HN