Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 Architecture Overview and Notes

sebastianraschka.com

Post reply on HN
Print view View original

101–110 of 125 posts

Re: Kimi K3 Architecture Overview and Notes

#101

Post by cma »

itkovian_ wrote: post #22
Ilaurens wrote: post #21

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead." It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

The paper says:

"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."

Re: Kimi K3 Architecture Overview and Notes

#102

Post by Aldipower »

rurban wrote: post #92
dools wrote: post #91

They may have all this amazing architecture but Kimi has been super dumb recently. I reckon they’re under compute pressure and quantising to stay afloat. I was a heavy k2.5/2.6 user earlier in the year and built serious features with it, but even k3 now does stupid shit like fail tool calls and get stuck in endless thought trains. K2.6 was spinning its wheels on a problem for over 10 minutes today and then deepseek v…

Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.

I have a lot of problems with Opus 5 in the last days. So, maybe the problem is to expect reliability from something probabilistic?

Re: Kimi K3 Architecture Overview and Notes

#103

Post by rurban »

dools wrote: post #93
rurban wrote: post #92

Earlier quoted context omitted.

Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.

I’ve had no issues with deepseek at all. Maybe it’s just luck of the draw

Good to hear. Most likely one of those monthly hickups, destroying main. Needed lots of reverts, and manual fixing. If in a branch I wouldn't care that much.

Re: Kimi K3 Architecture Overview and Notes

#104

Post by davidguetta »

nl wrote: post #67
mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

the beginning of the arguments was good but the "because of randomness" is wild

Re: Kimi K3 Architecture Overview and Notes

#105

Post by nvme0n1p1 »

embedding-shape wrote: post #100
nvme0n1p1 wrote: post #95

Earlier quoted context omitted.

> You have a very interesting understanding of "reproducibility", I'll give you that :) He was talking about two different things, hence the parentheses. The architecture is reproducible, not the model weights.

> The architecture is reproducible But what's the point of even saying that? Of course it is, otherwise how is it supposed to run in the runtimes? You cannot release model weights that others can run, without also releasing the model architecture, it's in the code at the very least...

Because for other specs it's not the case.

The K3 arch can be implemented from the spec.

A PDF reader can not be implemented from the spec.

Re: Kimi K3 Architecture Overview and Notes

#106

Post by [deleted] »

Giho wrote: post #97

I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix: St =St−1 +βt (vt −St−1 kt )ktT The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss. Interesting what other labs comes up with from this.

[deleted]

Re: Kimi K3 Architecture Overview and Notes

#107

Post by pcmasterr »

eru wrote: post #84
desterothx wrote: post #81

Earlier quoted context omitted.

not really, compiling isnt a heuristic problem, it has a lot less randomness involved

Depends on your compiler. You could have a compiler that deliberately uses randomised algorithms. They are often faster and easier to understand and write. Though in practice you can get all the benefits of both determinism and (that kind of) randomisation by using a PRNG and saving the seed you are using. It's an open question roughly on par with P vs NP whether true randomisation is ever necessary, or whether PRNGs…

Also, most (optimizing) compilers ”optimize” the code for a fixed amount of time, leading to better optimized binaries on faster computers. That’s why developers should have as fast computers money can buy!

Re: Kimi K3 Architecture Overview and Notes

#108

Post by pcmasterr »

eru wrote: post #85
marcyb5st wrote: post #83

Earlier quoted context omitted.

The exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.

> Should have minimal impact on the final outcome, [...] I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.

Reproducibility isn’t something that has been considered desirable in many pipelines until recent years.

Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.

Re: Kimi K3 Architecture Overview and Notes

#110

Post by negativezone »

igravious wrote: post #75
gboss wrote: post #61

Anybody getting the result that Kimi 3 is more expensive than Opus 5 or Sol on Cursor? Pretty sure Kimi 3 sucked up a good chunk of my ultimate plan in a few prompts. Anyone have any tools or ways to understand per model usage towards cursor subscriptions? I know there are alternatives to cursor just haven’t made the move yet. (Edit spelling)

The assertions doing the rounds that Kimi K3 and GLM 5.2 are way cheaper than Claude/GPT are not true -- DeepSeek V4 Pro is a lot cheaper but K3 and 5.2 ain't. Turns out that you actually have to fork out some cash for frontier-esque models, be they Chinese or American. Hope that helps. Source: my bank balance

is it cheaper for medium scale folks? not everyone needs to go big on ai subscriptions and a solid, reliable midrange option wouldn't go amiss
Post reply on HN