Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 Architecture Overview and Notes

sebastianraschka.com

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Kimi K3 Architecture Overview and Notes

#111

Post by rglover »

igravious wrote: post #76
rglover wrote: post #40

Just tried K3 out for the first time today and it's a legitimate threat. Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]). [1] https://graybearding.bearblog.dev/kimi-k3-is-insane/

threat? that's a weird way of describing a near frontier open weights un-crippled useful coding buddy do you work for OpenAI or Anthropic per chance?

I work for myself. A quick check of my profile here would make that clear.

And yes, it's a legitimate threat to frontier models. Both K3 and the distillation approach.

Re: Kimi K3 Architecture Overview and Notes

#112

Post by monocasa »

nl wrote: post #67
mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations). (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the da…

There's more to open source than reproducibility. For instance introspection which is even more important with weights since detection of backdoors in models is NP hard IIRC.

Re: Kimi K3 Architecture Overview and Notes

#114

Post by ModelForge »

mickael-kerjean wrote: post #43

Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?

Good question, it's 100%. I.e., the developers usually share a reference implementation with e.g., Hugging Face transformers to load their weights, and from there on you can read the code and, if you have time, reimplement and check everything. It's actually a great learning exercise where you can self-check whether you reimplemented it correctly by comparing the LLM outputs to the reference implementation. Made a video about that workflow a while back here: https://www.youtube.com/watch?v=TXzQ7PGpO6w

Re: Kimi K3 Architecture Overview and Notes

#115

Post by tomrod »

rsolva wrote: post #26
thatsgcasey wrote: post #18

Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack

This seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!

His was the only Packt book (on ML, circa 2019 IIRC) that I have never regretted purchasing. Clear thinker, attention to details, and clean coder. Highly recommend.

Re: Kimi K3 Architecture Overview and Notes

#117

Post by KronisLV »

nmfisher wrote: post #68
ATMLOTTOBEER wrote: post #48

Earlier quoted context omitted.

Agreed. Dropped my $200/mo Claude I was saturating for the $100/mo moonshot sub and have no complaints Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…

How much usage do you get out of the $100 Moonshot plan? I haven't heard great things.

For me it got exhausted faster than I'd like, subjectively less value than the Anthropic 100 USD subscription, BUT their 5 hour limits are very generous, whereas I keep bumping into Anthropic's all the time: https://blog.kronis.dev/blog/kimi-k3-is-out-is-anthropic-don...

Their 200 USD tier is pretty good and I might move to them because their annual discounts also make it even cheaper. I dislike the way how Opus/Fable write and Kimi has less of that slop writing problem, but also ends up feeling kinda slower (especially on Max reasoning), technical abilities are not 100% the same but close (better in some tasks, worse on others).

Re: Kimi K3 Architecture Overview and Notes

#118

Post by KronisLV »

Aldipower wrote: post #102
rurban wrote: post #92

Earlier quoted context omitted.

Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.

I have a lot of problems with Opus 5 in the last days. So, maybe the problem is to expect reliability from something probabilistic?

Might be a bit more than just noise: https://marginlab.ai/trackers/claude-code-historical-perform...

Edit: well, crap.

  Model overloaded retrying (6/10) 1m 40s
Thanks, Anthropic.

Re: Kimi K3 Architecture Overview and Notes

#119

Post by NooneAtAll3 »

desterothx wrote: post #81
spider-mario wrote: post #77

Earlier quoted context omitted.

Isn’t this a bit like saying “ ‘open object files are not open source’ is a complete misunderstanding” because “You can’t rebuild the executable from scratch, but you can’t do this even with the source code anyway (because of build nondeterminism / compiler versions / etc.)”?

not really, compiling isnt a heuristic problem, it has a lot less randomness involved

there's a reason reproducible builds are a thing and most compilations aren't

Re: Kimi K3 Architecture Overview and Notes

#120

Post by eru »

pcmasterr wrote: post #108
eru wrote: post #85

Earlier quoted context omitted.

> Should have minimal impact on the final outcome, [...] I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.

Reproducibility isn’t something that has been considered desirable in many pipelines until recent years. Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.

Reproducing a stationary probability distribution that subsequent runs draw from is also a kind of reproduction. And presumably for ffmpeg you can fix the random seed? Tell me more about that design, please.
Post reply on HN