Kimi K3 Architecture Overview and Notes • Hacker News Board

Re: Kimi K3 Architecture Overview and Notes

#121

Post by eru » Wed, Jul 29, 2026, 11:13 PM UTC

pcmasterr wrote: post #107

eru wrote: post #84

Earlier quoted context omitted.

Depends on your compiler. You could have a compiler that deliberately uses randomised algorithms. They are often faster and easier to understand and write. Though in practice you can get all the benefits of both determinism and (that kind of) randomisation by using a PRNG and saving the seed you are using. It's an open question roughly on par with P vs NP whether true randomisation is ever necessary, or whether PRNGs…

Also, most (optimizing) compilers ”optimize” the code for a fixed amount of time, leading to better optimized binaries on faster computers. That’s why developers should have as fast computers money can buy!

> That’s why developers should have as fast computers money can buy!

I don't see the connection? Most local builds are done with debugging on and optimisation turned off anyway.

And what you deliver to your customers is usually something you produce on your CI/CD server, not what's on any developer's machine.

(And if you want reproducible builds https://en.wikipedia.org/wiki/Reproducible_builds you can't optimise for a specific wall clock time.)

> Also, most (optimizing) compilers ”optimize” the code for a fixed amount of time, leading to better optimized binaries on faster computers.

That’s why you should give your developers computers that have slow clocks!

Re: Kimi K3 Architecture Overview and Notes

#122

Post by eru » Wed, Jul 29, 2026, 11:15 PM UTC

alightsoul wrote: post #89

eru wrote: post #82

Earlier quoted context omitted.

> (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!)) They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomn…

Do AI labs even keep track of seeds? They don't even keep the checkpoints most of the time.

It's moot in practice, because there's additional sources of randomness from executing in parallel.

Though in principle saving random seeds is a lot less hassle than keeping entire checkpoints around: your random seeds would fit on a floppy disk or even a tweet. The checkpoint is basically as big as the model.

Re: Kimi K3 Architecture Overview and Notes

#123

Post by nl » Sat, Aug 01, 2026, 7:17 AM UTC

FinchNova12 wrote: post #73

nl wrote: post #67

Earlier quoted context omitted.

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

I agree that if you have the weights you can use/train a model with the same architecture, and that you won't get the exact weights on your own due to randomness. But isn't data an extremely important part of your ability to effectively train/finetune? It might be much harder to get close to the level of the open weight model if you don't have the data that made it, which is why I think the open weights vs open sourc…

You fine-tune the open-weight model without access to the original weights.

If you want to train it from scratch you need data, yes. But presumably if you are doing that there is a reason you want to do it.

You lose nothing without access to the original data - you can do every single modification without it.

That is unlike open source where you (mostly) need to source code to modify it beyond what the original designed originally thought.

Re: Kimi K3 Architecture Overview and Notes

#124

Post by nl » Sat, Aug 01, 2026, 7:21 AM UTC

embedding-shape wrote: post #100

nvme0n1p1 wrote: post #95

Earlier quoted context omitted.

> You have a very interesting understanding of "reproducibility", I'll give you that :) He was talking about two different things, hence the parentheses. The architecture is reproducible, not the model weights.

> The architecture is reproducible But what's the point of even saying that? Of course it is, otherwise how is it supposed to run in the runtimes? You cannot release model weights that others can run, without also releasing the model architecture, it's in the code at the very least...

[dead]

Re: Kimi K3 Architecture Overview and Notes

#125

Post by nl » Sat, Aug 01, 2026, 7:22 AM UTC

davidguetta wrote: post #104

nl wrote: post #67

Earlier quoted context omitted.

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

the beginning of the arguments was good but the "because of randomness" is wild

Why?

Have you ever tried reproducing even a small neural network exactly if you train on GPUs on more than one machine? I have and it is pretty close to impossible, and I'd argue actually impossible at scale.