291–300 of 742 posts
[flagged]
Yes they distill, but if you think you can trivially get a frontier-level model by "just" distilling from Claude's public API. you fundamentally do not understand the amount of work that goes into a modern post-training stack. Without even talking about the fact that any distillation that was done was on Opus, as the timeline of Mythos/Fable vs Kimi 3 release dates just do not match in any plausible way. If you want…
Earlier quoted context omitted.
It matters because the closed-source frontier labs spend lots of money on human data (RLHF / RLAIF with human oversight). Moonshot is accused of circumventing these costs. Frontier labs add research costs into their inference pricing. If the market doesn't permit them to sustain sufficient pricing to have a positive cash flow, then their business prospects become weaker and they risk insolvency. Furthermore, other le…
Chatgpt routinely cites and uses papers I don't have access to because they're behind a paywall. I don't think OpenAI is paying for all that copyright. That's in my opinion way more serious.
Earlier quoted context omitted.
Perhaps even more importantly, the current frontier LLM models are self-admittedly the product of enormous quantities of copyright infringement and even less savory inputs, so calling them out for distilling the fruit of that tainted tree reads as highly hypocritical at best.
While I agree on a moral level, I think there is a distinction to be made. Training a SOTA model takes a huge amount of resources and expertise so the people doing the training are adding a lot of value along the way. I think this is much less true for distillation (which is kind of the whole point). ed: to clarify, I totally agree that a huge chunk of the value in LLMs is coming from the source material. My point wa…
Earlier quoted context omitted.
They're probably going for the national security/domestic manufacturing angle. > Aren't consumers benefiting from this practice by getting better cheaper models as a result? Aren't consumers benefiting from cheap chinese batteries, EVs, and drones?
I am not sure if this is implicitly part of the point you meant to make, but I just wanted to point out for those who aren't aware that Chinese EVs and drones are both banned in the US on precisely the (vague) grounds you mention. The drone ban is more recent and nominally only affects new models that haven't yet received FCC certification, but the outcome if nothing changes will be that American consumers lose acces…
I mean not really. A quadrocopter is a remarkably simple thing made out of extremely generic parts: 4 DC motors (and ESCs), a radio, a computer, a battery and an inertial measurement unit. Anybody with a rudimentary amount of electronics knowledge can build one, the components are extremely widely used. The most unique parts about them are the frame and propellers, which are pretty easy to fabricate.
Earlier quoted context omitted.
They're probably going for the national security/domestic manufacturing angle. > Aren't consumers benefiting from this practice by getting better cheaper models as a result? Aren't consumers benefiting from cheap chinese batteries, EVs, and drones?
> Aren't consumers benefiting from cheap chinese batteries, EVs, and drones? Depends on what country you live in I suppose, but likely a spectrum of yes than any outright no. For example, Chinese EVs are using a different battery chemistry and not putting demand pressure on the more expensive chemistry western manufacturers use
They are a petroleum products importer, so it’s a big win for them.
Here's a site that asks the same questions to 22 models and compares how similar their responses are. https://typebulb.com/u/lab/you-re-relatively-right/full According to these results GLM 5.2 is very similar to Google Gemini and Kimi K3 is very similar to Fable 5. The American frontier labs are not similar to each other.
Interesting. This dooes lend credence to the distillation idea. Good for them!
Mendel doesn't get a cut every time somebody uses the principles of heritability he discovered, and Einstein's family aren't getting royalties if you compute relative speeds. I think the frontier labs should expect to be treated more like scientists than artists in this regard.
Kimi K3 was released July 16, Fable ban was lifted on July 1 but access was still limited. How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies? I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies
[flagged]
Yes they distill, but if you think you can trivially get a frontier-level model by "just" distilling from Claude's public API. you fundamentally do not understand the amount of work that goes into a modern post-training stack. Without even talking about the fact that any distillation that was done was on Opus, as the timeline of Mythos/Fable vs Kimi 3 release dates just do not match in any plausible way. If you want…
Earlier quoted context omitted.
This is why I don't give a shit that this is happening. It's actually kind of funny to me.
Unless you’re from mainland China, you should.