Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Kimi K3 Moment

stephen.bochinski.dev

Post reply on HN
Print view View original

341–350 of 644 posts

Re: The Kimi K3 Moment

#341

Post by tristanj »

overfeed wrote: post #322
tristanj wrote: post #305

Earlier quoted context omitted.

3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million. There are tens of thousands of accounts funneling Claude sessions into Chinese labs https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens... They are used for post-training, i.e. calibrating the model to understand and use tools/command line more effectively.

> 3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million. That's an increase of only a single order of magnitude, increasing my estimate of exfiltrated tokens from 0.05 to 0.15 trillion - a far cry from the 15 trillion required. > They are used for post-training Possibly - it may be too much data for post-training, unless further curation was d…

You're conflating pre-training data volume with post-training data volume.

Nobody is suggesting Moonshot used 15 trillion tokens of Claude data to pre-train a base model from scratch. That would be impossible and nonsensical.

This is entirely about distillation, which happens during post-training (alignment and SFT). Here, datasets are measured in millions or billions of tokens, not trillions. 50 billion Claude tokens is far, far than enough to copy Claude's reasoning logic, writing style, and tool-use ability to the pre-trained base model.

> However, this is not distillation

I don't understand how you're so caught up on the term "distillation". Distillation is using a larger model's outputs to train a (weaker) student model. Which is exactly what's happening. It's a standardized term that has been in use for a decade.

Re: The Kimi K3 Moment

#342

Post by rayiner »

warshinder wrote: post #340
rayiner wrote: post #336

Earlier quoted context omitted.

The desire to accuse China of just copying is like 20 years out of date. It’s been wrong since some people on HN were in diapers. People are going to be gobsmacked when, in our lifetime, China becomes a world power comparable to the U.S. Probably still poorer per capita, but at Spain/Italy levels, not third world country levels. And they’ll be shocked at the implications of that on the world economy, migration patter…

So the efficient market hypothesis is wrong?

How is the efficient market hypothesis applicable here?

Re: The Kimi K3 Moment

#343

Post by sm-silversight »

warshinder wrote: post #340
rayiner wrote: post #336

Earlier quoted context omitted.

The desire to accuse China of just copying is like 20 years out of date. It’s been wrong since some people on HN were in diapers. People are going to be gobsmacked when, in our lifetime, China becomes a world power comparable to the U.S. Probably still poorer per capita, but at Spain/Italy levels, not third world country levels. And they’ll be shocked at the implications of that on the world economy, migration patter…

So the efficient market hypothesis is wrong?

What do you mean, I don't follow.

Also, yes, often.

Re: The Kimi K3 Moment

#344

Post by kamranjon »

tristanj wrote: post #282
kamranjon wrote: post #248

Earlier quoted context omitted.

The fact that API based distillation is even a conversation right now makes me feel like the U.S. has their heads so far in the sand that it’s not really excusable. These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs. Setting aside the fact that it doesn’t make any feasible sense to do API dis…

There's little doubt that Kimi K3 was distilled off Claude. Anthropic stated in February that Moonshot AI (the creator of Kimi) distilled ~3.4 million exchanges from Claude models, as explained in their press release https://www.anthropic.com/news/detecting-and-preventing-dist...

It’s so funny to me that Anthropic can make claims like this one with zero evidence provided.

DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.

Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.

Re: The Kimi K3 Moment

#346

Post by tristanj »

kamranjon wrote: post #344
tristanj wrote: post #282

Earlier quoted context omitted.

There's little doubt that Kimi K3 was distilled off Claude. Anthropic stated in February that Moonshot AI (the creator of Kimi) distilled ~3.4 million exchanges from Claude models, as explained in their press release https://www.anthropic.com/news/detecting-and-preventing-dist...

It’s so funny to me that Anthropic can make claims like this one with zero evidence provided. DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe. Chinese labs might not have the funding of labs l…

There's reproducible evidence of Kimi K3 spontaneously identifying itself as Claude https://x.com/denisewu/status/2077984660211269870

This behavior is exactly what you'd expect from a model distilled from Claude.

Someone even took the time to analyze Kimi's ambiguous identity, in great detail: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/...

And there's an entire Reddit thread discussing this https://www.reddit.com/r/LocalLLaMA/comments/1m2w5ge/did_kim...

That doesn’t prove Anthropic’s specific 3.4m-session allegation, but calling it “zero evidence” is no longer credible.

Kimi K2.5 was worse in a hilarious way, it identified itself as Claude and referenced Anthropic's Constitutional AI as some of its guiding principles https://huggingface.co/moonshotai/Kimi-K2.5/discussions/38

Re: The Kimi K3 Moment

#348

Post by tristanj »

anon373839 wrote: post #223
nickysielicki wrote: post #33

Regardless of whether they achieved parity via distillation, or whether they got here via independently constructing a model from scratch, it was always going to end this way for the frontier American labs. Distillation “attacks” are not attacks. The frontier labs “distilled” all existing human written knowledge into their models, there was always going to be a second class lab that would distill that model into a ch…

I strongly agree with the premise that distillation is not an “attack”. But that said: K3 is not a distilled version of Fable or Sol. Fable has been barely available and Sol was just released! Moreover, K3 is superior to both models in some domains, according to user scoring on the Arena. API distillation can’t give you these results anyway. All it is useful for is bootstrapping RL in new domains to get past the “col…

API distillation doesn't have to explain all of K3's capabilities for it to have happened. Kimi K3 reproducibly identifies itself as Claude: https://x.com/denisewu/status/2077984660211269870

This behavior is exactly what you'd expect from a model distilled from Claude.

There's a detailed analysis of K3's ambiguous identity here: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/...

This analysis observed K3 identifies itself as Claude approximately 15% of the time.

K3 reproduces Claude's correct current model id, which the real Claude models themselves do not emit. This suggests K3 was trained on Claude data labeled with deployment metadata (API logs, tagged synthetic data), rather than Claude's chat outputs.

And there's an entire Reddit thread discussing Kimi's similarities with Claude https://www.reddit.com/r/LocalLLaMA/comments/1m2w5ge/did_kim...

This analysis shows K3 and Opus/Fable have unexpected correlated outputs https://typebulb.com/u/lab/you-re-relatively-right/full

Re: The Kimi K3 Moment

#349

Post by staticman2 »

tristanj wrote: post #346
kamranjon wrote: post #344

Earlier quoted context omitted.

It’s so funny to me that Anthropic can make claims like this one with zero evidence provided. DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe. Chinese labs might not have the funding of labs l…

There's reproducible evidence of Kimi K3 spontaneously identifying itself as Claude https://x.com/denisewu/status/2077984660211269870 This behavior is exactly what you'd expect from a model distilled from Claude. Someone even took the time to analyze Kimi's ambiguous identity, in great detail: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/... And there's an entire Reddit thread discussing this https://w…

> This behavior is exactly what you'd expect from a model distilled from Claude.

This is not at all what I would expect because it's trivial to change the training data to replace Claude with Kimi. In fact I'd argue it's almost certainly not saying that due to distillation.

Re: The Kimi K3 Moment

#350

Post by tristanj »

staticman2 wrote: post #349
tristanj wrote: post #346

Earlier quoted context omitted.

There's reproducible evidence of Kimi K3 spontaneously identifying itself as Claude https://x.com/denisewu/status/2077984660211269870 This behavior is exactly what you'd expect from a model distilled from Claude. Someone even took the time to analyze Kimi's ambiguous identity, in great detail: https://github.com/rgreenblatt/which_claude_is_k3/blob/main/... And there's an entire Reddit thread discussing this https://w…

> This behavior is exactly what you'd expect from a model distilled from Claude. This is not at all what I would expect because it's trivial to change the training data to replace Claude with Kimi. In fact I'd argue it's almost certainly not saying that due to distillation.

I encourage you to review the links before committing to a position. The writeup on K3's anomalous trans-model identity is very comprehensive.

K3 reproduces Claude's internal model identifier when prompted, something which the real Claude models themselves do not emit. This is highly suggestive that K3 was trained on Claude metadata (API logs, tagged synthetic data), rather than Claude's chat outputs.

And it's well documented that Chinese labs are buying large amounts of raw Claude metadata https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens...

Post reply on HN