Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“We have information that Moonshot distilled Fable for the development of K3”

twitter.com

Post reply on HN
Print view View original

51–60 of 742 posts

Re: “We have information that Moonshot distilled Fable for the development of K3”

#51

Post by epolanski »

throwa356262 wrote: post #12

Kimi K3 was released July 16, Fable ban was lifted on July 1 but access was still limited. How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies? I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies

This is BS to pressure politicians.

Even an openai's guy (head of something made up) called bs on the idea you can train something like k3 by distillation.

Anybody I know who works in LLM research says that distillation is either useless or merely useful in post training to show "correct" behavior.

And even then you don't get a competing model, if RL on good prompts was that useful, all labs would've long skyrocketed in capabilities just by looping on increasingly better prompts, yet that doesn't work.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#52

Post by Gregaros »

hobonation wrote: post #33
throwa356262 wrote: post #12

Kimi K3 was released July 16, Fable ban was lifted on July 1 but access was still limited. How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies? I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies

I sort of did it. I got Fable to set up an AI system with better and better prompts within my app. At the end of it, Fable made me an AI system that works well enough that my users don't need Fable. Obviously, it's not K3 level. But Fable did just put itself out of a job in this case.

You did not distill Fable. Relevantly, what you did provides no evidence contrary to the parent’s assertion that Moonshot did not have time to distill Fable.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#53

Post by [deleted] »

sosodev wrote: post #31
kouteiheika wrote: post #25

Assuming they did then they surely paid for them, which makes it "not stealing". Am I also "stealing proprietary U.S. technology" by harvesting my Claude chats from my `.claude` directory and training a bunch of models on them? That said, I doubt the "they distilled Fable" is the reason why K3 is as good as it is, considering the timelines involved, and that Anthropic hides thinking traces, and their overly aggressiv…

Model distillation can't be stealing at all if you rationally apply copyright law to it. Anthropic is not deprived of Fable so there is no theft. At best it would be infringement, but even that might not hold up in the courts given the current position that model outputs can't be subject to copyright.

[deleted]

Re: “We have information that Moonshot distilled Fable for the development of K3”

#54

Post by epolanski »

mattrighetti wrote: post #29

Is distillation something we have to live with or are there ways to prevent it?

If it was genuinely useful, we would've long reached the point where you train a model on a previous one's output in an ever improving loop.

But this doesn't actually work.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#56

Post by runarberg »

cwmoore wrote: post #20
superloika wrote: post #9

I think they deserve, by Justice, to have their models pillaged and raped, just like they did to the internet. They didn't ask for permission when they took the entire of the internet, after all, and given their behaviour is nefarious, it's of Justice that they receive nefarious treatment by others, including chinese AI labs. The Chinese are not gonna deterred, but the posturing by the Americans is so blatantly hypoc…

I still believe taxing the bots, and implementing actual UBI, would address both problems.

And I believe a socialist revolution in international solidarity of the working classes against our exploiters the capitalist owning class would address both problems as well (and more), but in the meantime I’ll be happy whenever I spot poetic justice in the wild.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#57

Post by make3 »

hobonation wrote: post #33
throwa356262 wrote: post #12

Kimi K3 was released July 16, Fable ban was lifted on July 1 but access was still limited. How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies? I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies

I sort of did it. I got Fable to set up an AI system with better and better prompts within my app. At the end of it, Fable made me an AI system that works well enough that my users don't need Fable. Obviously, it's not K3 level. But Fable did just put itself out of a job in this case.

Distillation requires training

Re: “We have information that Moonshot distilled Fable for the development of K3”

#58

Post by epolanski »

sosodev wrote: post #22
alightsoul wrote: post #7

how is it possible to distill fable only a month after its release? maybe they are confusing opus with fable.

A month seems plenty long enough. They're not rebuilding the entire model from scratch. It's just getting Fable to act as a teacher model for some of the final reinforcement learning on the base that Kimi already had.

I think that would also be a bad idea, as all models opus 4.6 got increasingly smarter, but also crappier at following instructions or genuinely assisting.

They just try to figure out what the goal is and hyper focus on solving it.

Heh, even just telling fable don't commit doesn't work half the times, let alone more complex instructions.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#59

Post by gruez »

warkdarrior wrote: post #41
bradfa wrote: post #37

I can understand that the AI labs might care about other labs distilling their models as it can eat into their competitive advantage, but do consumers care at all? Aren't consumers benefiting from this practice by getting better cheaper models as a result?

I demand my right to pay 3x more for AI access. cf https://www.reddit.com/r/codex/comments/1uyj6pq/kimi_k3_is_1...

But nobody really pays the 3x (ie. api) rates, except for enterprises. Everyone else are using the consumption plans, which are heavily discounted[1], possibly cheaper than even the chinese models, which don't do consumption plan discounts. Even in your linked reddit thread, the OP agreed with this sentiment.

[1] https://x.com/SemiAnalysis_/status/2064815044085318040

Post reply on HN