81–90 of 141 posts
If you want to believe that the success of Kimi is about distillation attacks, ignore this.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Earlier quoted context omitted.
Agreed. I think when it comes to light that Claude has been known to say “I’m DeepSeek” that everyone has had their hand in that cookie jar. Moreover, paying for API calls hardly seems like an attack; ToS violation to be certain but not in the same category of law as criminal activity like hacking.
wait, is there evidence of this? I've not observed it. It sounds like the kind of thing that I want to be true because it would be hilarious but that makes me suspicious.
Earlier quoted context omitted.
Google is apparently taking a different stance and offering distillation as a paid product https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
You don't get to take distilled model home, it all stays with Google. It's "optimize your costs in our garden" product.
I recommend Fireworks as an alternative
Earlier quoted context omitted.
This is actually a well-known phenomenon in ML, called "The Bitter Lesson". > One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning. The full essay is worth a read, it's pre…
[flagged]
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791 ), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
Is it just me or does this read like a re-implementation of LSTMs?
Earlier quoted context omitted.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint. No, it's perfectly reasonable once you get down to reality. China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny. We don't have t…
why should western AI companies be held to different standards than Chinese ones? Neither of them are your buddy.
So distillation, among other things, allows Chinese labs to indirectly benefit from these arrangements at no cost to them.
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled? Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture? It's unintuitiv…
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset: https://en.wikipedia.org/wiki/Grokking_(machine_learning) High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima: https://www.youtube.com/watch?v…
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled? Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture? It's unintuitiv…
And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and memorize every combination. They call this "slingshot generalization".