This looks to be more behavior based, not logit based? What is the actual claim?
It seems to boil down to China bad, US good, but with tech people speculating so it's definitely valid
Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
11–20 of 73 posts
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#12All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
Does distilling actually violate any IP protection laws? Sure, it’s against their terms of service.
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#13All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#14All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
What of the costs for creating the data that was used to train the model being distilled?
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#15All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
An incredibly ironic comment.
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#16Stolen data was stolen. Oh no! Anyway.
Can you elaborate? This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet. Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizi…
I don't believe I understand your argument? Are you claiming a moral, legal or practical difference? Or are you saying that Anthropic spending resources on training an LLM is somehow different from an author spending resources on writing a book?
In any case, I doubt Kimi was trained without "stealing" the same data. Assembling all of your training data from Claude responses seems infeasible. It's much more likely that Kimi's base model was trained similarly to any other base model, with terabytes of data from all imaginable sources. Then the model was fine-tuned with "high-quality" data, followed by reinforcement learning. Throwing in lots of chat transcripts from other chatbots into the "high-quality" dataset would be expected, and is done to some degree by everyone, but maybe a lot more for Kimi. And likely they did a lot of reinforcement learning against the Claude API
The model would exist without Claude, it just wouldn't be nearly as coherent or smart
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#17Stolen data was stolen. Oh no! Anyway.
Can you elaborate? This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet. Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizi…
All ML/AI models are comparable to some form of compression(al beit lossy) of information and in this case copyrighted information. The OP is pointing to this as stolen data(by all the companies that started with pre trained models)
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#18Earlier quoted context omitted.
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
What of the costs for creating the data that was used to train the model being distilled?
At least 1.5B by stealing books, per a recent ruling.
I don’t feel sorry for the model companies
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#19Stolen data was stolen. Oh no! Anyway.
Can you elaborate? This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet. Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizi…
So what?
If training on copyrighted data without authors consent is ok then distilling is ok as well.
Re: Writing fingerprint analysis of responses reveals Kimi's similarity to Claude
#20All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
I'm going to hope this was sarcasm and if it is, it's great.