101–110 of 141 posts
Earlier quoted context omitted.
why should western AI companies be held to different standards than Chinese ones? Neither of them are your buddy.
They shouldn't be. But the point is, they are. OpenAI and Anthropic are paying many rights holders for access to their data (reddit, NYT, etc.). So distillation, among other things, allows Chinese labs to indirectly benefit from these arrangements at no cost to them.
If you want to believe that the success of Kimi is about distillation attacks, ignore this.
False dichotomy right? Are Chinese labs impressively innovating? Clearly. However this doesn’t rule out possible gains due to distillation. I don’t know the degree of the latter but both things could certainly be true.
Earlier quoted context omitted.
I'd kindly suggest that we could also stop calling them "distillation attacks ".
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
Earlier quoted context omitted.
I'd kindly suggest that we could also stop calling them "distillation attacks ".
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
A model is teaching another here
Everyone wins
Earlier quoted context omitted.
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
Earlier quoted context omitted.
False dichotomy right? Are Chinese labs impressively innovating? Clearly. However this doesn’t rule out possible gains due to distillation. I don’t know the degree of the latter but both things could certainly be true.
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints. This is just awesome.
Earlier quoted context omitted.
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen). Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint. No, it's perfectly reasonable once you get down to reality. China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny. We don't have t…
Anthropic is currently angling for a "IP restrictions for China but not for me" la-la land scenario. If they were instead arguing for either of the choices you said, it wouldn't be hypocritical.
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791 ), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
Is it just me or does this read like a re-implementation of LSTMs?
The nvidia version is heavy to compute (common problem with RNN and LSTM, also noted in the paper). Moonshot's Kimi K3 replaces part of it with some function that performs better.
I dont understand the details but, that in itself might a pretty big contribution.
Anyway, I'm amazed how fast these companies improve each other's ideas and put them in new products.
Earlier quoted context omitted.
Google is apparently taking a different stance and offering distillation as a paid product https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
But nobody wants to distill Google's models, Gemini is really bad.