Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“We have information that Moonshot distilled Fable for the development of K3”

twitter.com

Post reply on HN
Print view View original

671–680 of 742 posts

Re: “We have information that Moonshot distilled Fable for the development of K3”

#671

Post by dijksterhuis »

throw10920 wrote: post #621
bluegatty wrote: post #583

Earlier quoted context omitted.

I'm not attacking you, you don't have to defend yourself. I'm just nothing that HN rhetoric is contradictory. But this: "when i see a spade, i call it a spade." -> this is anti intellectual absolutism. If it were some true injustice, then fine, but that is clearly not the case. There is ample room to contemplate that even copyrighted works could be considers fair use as training material. "the corpos can get fucked a…

> I'm just nothing that HN rhetoric is contradictory. Periodic reminder that HN is not a collective or a singular entity and is actually a bunch of different people with different opinions. Often the people with the loudest opinions get upvoted to the top - and often the "side" represented at the top is different from thread to thread.

also, human beings themselves can be contradictory. as an example i will happily pirate films/tv shows, but refuse to do the same with music.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#672

Post by dijksterhuis »

bluegatty wrote: post #583
dijksterhuis wrote: post #572

Earlier quoted context omitted.

> it's hilarious to see how all these HN intellectuals, who on any other day are rabidly against most DRM / IP issues all of a sudden become private property absolutists. for the record, i've always been rabidly pro-copyright since i worked at a performing royalty organization (prs for music) circa 15 years ago, way before i joined hn. i don't use llms for that reason. > It's also sadly hypocritical to see all this r…

I'm not attacking you, you don't have to defend yourself. I'm just nothing that HN rhetoric is contradictory. But this: "when i see a spade, i call it a spade." -> this is anti intellectual absolutism. If it were some true injustice, then fine, but that is clearly not the case. There is ample room to contemplate that even copyrighted works could be considers fair use as training material. "the corpos can get fucked a…

[dead]

Re: “We have information that Moonshot distilled Fable for the development of K3”

#673

Post by HarHarVeryFunny »

bluegatty wrote: post #629
HarHarVeryFunny wrote: post #593

Earlier quoted context omitted.

No - you can't distill if what you are given doesn't have the thing in it that you want to distill out of it. I don't know why it's so important to you to use the word "distillation", but it's the wrong word to use. BTW OpenAI on twitter also said that Kimi 3 "cannot be explained away by distillation or anything like that". The timeline of how long it takes to train a model and when Fable was released don't even line…

Distillation is absolutely - and uncontroversially - a valid term for what is happening here. This isn't really a debate, I'm not making a fine point - just check with all of the various defintions of the term. Moreover - the 'reasoning traces' are not required for distillation at all. Finally - it's entirely possible for them to have used Fable for later stage fine tuning. It's fair to be skeptical of Anthropic (and…

Words have meaning - you cant just redefine them because you want to.

Are reasoning traces required for distillation? Well they are if what you are trying to distill is reasoning, such as coding expertise.

Do you need reasoning traces for "LLM as judge"? No, but it would be highly perverse to call that distillation when there is a more accurate name for it - LLM as judge.

If you want to call use of Anthropic's redacted model outputs in any fashion that violates their terms of service (using them them to help develop anything that competes with Anthropic) as "distillation" then I can't stop you, but it reduces their claims to a joke.

Finally, as noted, OpenAI (who are just as anti-Chinese as Anthropic) said that Kimi 3 can't be explained via distillation (even true distillation!!), or even "anything like it". But random internet guy, you, disagrees. OK.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#674

Post by dijksterhuis »

wasfgwp wrote: post #647
dijksterhuis wrote: post #572

Earlier quoted context omitted.

> it's hilarious to see how all these HN intellectuals, who on any other day are rabidly against most DRM / IP issues all of a sudden become private property absolutists. for the record, i've always been rabidly pro-copyright since i worked at a performing royalty organization (prs for music) circa 15 years ago, way before i joined hn. i don't use llms for that reason. > It's also sadly hypocritical to see all this r…

> two wrongs don't make a right There is no second “wrong” here. Model outputs are not copyrightable. I think that was already established? Or do you think that Anthropic should own all the code generated by Claude? Surely that would be somewhat problematic? If Anthropic feels that some of their customers are breaking their EULA (nothing to do with copyright infringement though) they are free to stop doing business w…

there's a difference between a moral wrong and a legal wrong. in a heavily simplified view, moral wrongs are usually decided in the eyes of the victims -- you did bad thing to me so i'm not going to talk to you anymore. legal wrongs are decided by courts -- you did a bad thing so this court has decided you're not allowed to talk to that person anymore.

anthropic are essentially saying in this tweet they believe a moral wrong has been committed against them -- "unacceptable behaviour" etc.

plenty of people have been vocal about the fact anthropic have committed moral wrongs at scale in building the products in the first place, with the question of legal wrongs still being worked out.

so, two moral wrongs. legally, fuck knows.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#675

Post by amluto »

tristanj wrote: post #656
amluto wrote: post #653

Earlier quoted context omitted.

As far as I know, 100% of those Fable interactions would have had encrypted reasoning blocks, so distillation would be distinctly nontrivial even if the data were somehow available.

Encrypted reasoning traces don't prevent distillation. You only need the input prompts and final output responses to distill capabilities. As Anthropic explained in their distillation report (linked above), Moonshot AI already collected millions of session traces, covering: * Agentic reasoning and tool use * Coding and data analysis * Computer-use agent development * Computer vision Hiding the internal CoT blocks sto…

How would you propose to distill Fable-style input/output pairs without the CoT?

If you use them as SFT input, you’ll be trying to train a model to predict the post-reasoning output without any reasoning, and this seem very unlikely to work at all with the size of model that Kimi produced and the complexity of Fable’s output. You can’t really “RL” with them because they would be so far off policy that there would be nothing to reinforce. I suppose you could feed input/output pairs to a teacher model and attempt to generate reasoning traces, but it seems like some wishful thinking would be required to get anything even close to as good as Kimi K3 out.

Maybe Kimi used these traces to generate RL gym-style problems and somehow produced an evaluator based on the outputs? They would not have had a lot of time in which to do this, and the learning style would not even remotely resemble that which Anthropic used to train Mythos/Fable in the first place.

But what do I know? I’m not an expert here.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#676

Post by HarHarVeryFunny »

throw10920 wrote: post #620
HarHarVeryFunny wrote: post #571

Earlier quoted context omitted.

You're being too charitable to Anthropic, and assuming that the way they are abusing the word "distillation" has some real meaning here. It doesn't. Anthropic's models simply do not give you their reasoning output - they give a sanitized "summary" instead, for this exact reason, so that the output is not useful to anyone who might want to use it for training. You can't distill what you are not given - simple as that.…

> Anthropic's models simply do not give you their reasoning output - they give a sanitized "summary" instead, for this exact reason, so that the output is not useful to anyone who might want to use it for training. This is just straight-up factually false. The output of a reasoning model is immensely valuable even without the sanitized summary of the reasoning process - that Anthropic's service does expose to you, ma…

> The output of a reasoning model is immensely valuable even without the sanitized summary of the reasoning process - that Anthropic's service does expose to you, making it even more valuable.

Useful for what is the question. Nobody is debating whether the outputs of LLMs are valuable.

Given that Anthropic have redacted their true reasoning, and replaced it with a "summary", specifically designed to be useless for distillation purposes, it would certainly be highly ironic if this summary was in fact "even more valuable" for that purpose as you are claiming!

Re: “We have information that Moonshot distilled Fable for the development of K3”

#677

Post by bigfishrunning »

SubiculumCode wrote: post #428
bigfishrunning wrote: post #342

Earlier quoted context omitted.

No, Machine Learning is an unfortunate name for a well documented process for creating black-box classifiers. The process is good, the name is not.

And what, to your mind, would classify something as learning? I assume that your position is not the hard "only humans/living creatures can learn"

Honestly, I'm not sure. But I do know that there is an entire field of cognitive science dedicated to understanding learning, and quite frankly it's in its infancy. Evidence of this is that every elementary school introduces new teaching techniques from time to time, and very rarely do they result in any benefit to the people who are doing the learning (more often they benefit consultants...).

However, the current process of "Machine Learning" (which is a semi-random parameter descent/evolutionary replacement process) is unlikely to be equivalent to the way people learn, because we aren't copying/competing/replacing our brain constantly. People are actually very good at learning, but our brain material replaces itself partially and relatively slowly (when compared to how a neural network is trained).

Re: “We have information that Moonshot distilled Fable for the development of K3”

#678

Post by delfinom »

nylonstrung wrote: post #298
throwa356262 wrote: post #12

Kimi K3 was released July 16, Fable ban was lifted on July 1 but access was still limited. How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies? I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies

If distillation truly is the cheat code they act like it is, then all the US and EU AI labs have no excuse for not having Fable-level models already

[flagged]

Re: “We have information that Moonshot distilled Fable for the development of K3”

#679

Post by gozucito »

remus wrote: post #631
gozucito wrote: post #464

Earlier quoted context omitted.

There is an even higher level of creativity in creating books, songs and all sorts of art used in model training though. That's your apparent blindspot. There is no world in which me vacuuming the entirety of human knowledge to make a genai model is ok but hoovering my model answers is not. The hypocrisy is stunning and risible. Now if you go and make a model based on purely synthetic data and not a single work made…

> There is an even higher level of creativity in creating books, songs and all sorts of art used in model training though. No argument here, I completely agree. > There is no world in which me vacuuming the entirety of human knowledge to make a genai model is ok but hoovering my model answers is not. I disagree with this though. Clearly LLMs owe a huge debt to everything that has come before, but surely you'd agree t…

Since this is HN Think of it like one of the GPL license for software.

It's ok for me to use your source code for free as long as I then let others also use my source code for free.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#680

Post by physicsguy »

linkregister wrote: post #301

Commenters are overlooking the significance of this information and posting emotional reactions based on perceptions of fairness or feelings of schadenfreude. The economic viability of Anthropic and OpenAI rely on their being able to charge more for model access than their R&D and inference costs. If the market price for SOTA model access drops below that level, then these businesses will have to decide whether to co…

> The United States government considers US supremacy in "AI" as a national security consideration.

They thought the same about SSL in the 1990s and the world didn't stop moving elsewhere.

Post reply on HN