Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“We have information that Moonshot distilled Fable for the development of K3”

twitter.com

Post reply on HN
Print view View original

181–190 of 742 posts

Re: “We have information that Moonshot distilled Fable for the development of K3”

#181

Post by mring33621 »

kevinqi wrote: post #178
himata4113 wrote: post #62

Does this matter? Distillation is not illegal by every definition of the word. There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them. Another example is that it appears that the upper limit of what you can do is ultimately dependent on people working on the model, otherwise grok would be a LOT more competitive pre…

I agree distillation isn't illegal; I also think Moonshot/Kimi is very impressive. But the more interesting question is whether labs like Moonshot can be a real competitor to OpenAI/Anthropic. If you can only play catchup (however quickly you do that), then you're never going to be at the frontier - I think that's why distillation matters.

People that think the Chinese are only able to copy western tech are in for a wakeup call.

Actually, that has already happened in many domains, it's just that most western people (USA especially) won't admit it.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#182

Post by ryandvm »

himata4113 wrote: post #62

Does this matter? Distillation is not illegal by every definition of the word. There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them. Another example is that it appears that the upper limit of what you can do is ultimately dependent on people working on the model, otherwise grok would be a LOT more competitive pre…

Boy I tell you, I am having an awful hard time summoning pity for the organizations that have themselves distilled all of humanity's knowledge into mysterious labor-market-masticating black boxes.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#184

Post by 4chandaily »

Seems to me like Moonshot is a paying customer, and if their business isn't worth the money Anthropic is charging, perhaps they should raise the price per token charged for it. Otherwise, I don't see why this is a story. "AI Company pays another AI Company for training data" just isn't that interesting.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#185

Post by mrhottakes »

JKCalhoun wrote: post #124
madduci wrote: post #55

So what is the issue here? Distilling is still fair, on the same level like Anthropic scraped copyright protected material for their training. So here robbers are blaming robbers? These claims are just pointless, everytime

I'm by no means taking the side of the AI companies, but it's possible that Anthropic "added value" to the data they harvested. Stealing that does seem kind of uncool. Regardless, it was always inevitable—will continue to happen.

So as long as Kimi added value to Fable, it's fine? Sounds good.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#186

Post by js8 »

throwa356262 wrote: post #89
epolanski wrote: post #51

Earlier quoted context omitted.

This is BS to pressure politicians. Even an openai's guy (head of something made up) called bs on the idea you can train something like k3 by distillation. Anybody I know who works in LLM research says that distillation is either useless or merely useful in post training to show "correct" behavior. And even then you don't get a competing model, if RL on good prompts was that useful, all labs would've long skyrocketed…

Dean Ball, "head of strategic futures" at openai. https://xcancel.com/deanwball/status/2078133895766114412#m

> AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape

I don't know what this guy thinks AI is, but this strikes me as delusional.

In my view, AI (LLM) is two things mixed together:

1. A reasoning engine on top of relatively rich fuzzy modal logic, implemented through variety of rules, which implement very common concepts.

2. A huge dictionary of words defined (with lot of detail) in the said logic, together with many known facts about them. Maybe bigger than Wikipedia.

Now, how on Earth do you want to gatekeep either of this? You can't gatekeep the 1st, logic of common sense, that's almost as difficult as gatekeeping a Turing machine (a concept of a computer). And gatekeeping the 2nd is ridiculous too, as it was built mostly from already published sources like a giant Wikipedia.

If anything, the opposite, to gatekeep AI is actually dystopian. It would mean end not only to right to compute, but also end of right to scientific knowledge.

(And I think, honestly, Chinese understand this. Trying to control-export AI makes as much sense as trying to control-export an English dictionary.)

Re: “We have information that Moonshot distilled Fable for the development of K3”

#187

Post by mrhottakes »

make3 wrote: post #64
madduci wrote: post #55

So what is the issue here? Distilling is still fair, on the same level like Anthropic scraped copyright protected material for their training. So here robbers are blaming robbers? These claims are just pointless, everytime

It's about the claim of whether these companies could develop a similarly powerful model without larger companies building their own first, which is an important point, and it's likely not the case. It's also about the larger companies explaining why they can't be as efficient, of course they can't, they're not just ripping the outputs of another model that someone else invested billions to train.

> they're not just ripping the outputs of another model that someone else invested billions to train.

True, they're simply ripping the inputs that humanity invested thousands of years and trillions of dollars to produce.

Re: “We have information that Moonshot distilled Fable for the development of K3”

#188

Post by andyfilms1 »

linkregister wrote: post #167
himata4113 wrote: post #62

Does this matter? Distillation is not illegal by every definition of the word. There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them. Another example is that it appears that the upper limit of what you can do is ultimately dependent on people working on the model, otherwise grok would be a LOT more competitive pre…

It matters because the closed-source frontier labs spend lots of money on human data (RLHF / RLAIF with human oversight). Moonshot is accused of circumventing these costs. Frontier labs add research costs into their inference pricing. If the market doesn't permit them to sustain sufficient pricing to have a positive cash flow, then their business prospects become weaker and they risk insolvency. Furthermore, other le…

Oh, so mass theft is okay as long as American companies are doing it

Re: “We have information that Moonshot distilled Fable for the development of K3”

#189

Post by spaceman_2020 »

linkregister wrote: post #167
himata4113 wrote: post #62

Does this matter? Distillation is not illegal by every definition of the word. There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them. Another example is that it appears that the upper limit of what you can do is ultimately dependent on people working on the model, otherwise grok would be a LOT more competitive pre…

It matters because the closed-source frontier labs spend lots of money on human data (RLHF / RLAIF with human oversight). Moonshot is accused of circumventing these costs. Frontier labs add research costs into their inference pricing. If the market doesn't permit them to sustain sufficient pricing to have a positive cash flow, then their business prospects become weaker and they risk insolvency. Furthermore, other le…

My parents put in countless hours and tens of thousands of dollars into raising me to the point where I could write an answer on StackOverflow

And OpenAI scraped and distilled that answer and gave me nothing

Post reply on HN