41–50 of 70 posts
Context: someone announced a Llama 3.1 70B fine tune with incredible benchmark results a few days ago. It's been a dramatic ride: - The weight releases were messed up: released Lora for Llama 3.0, claiming it was a 3.1 fine tune - Evals initially didn't meet expectations when run on released weights - The evals starting performing near/at SOTA when using a hosted endpoint - Folks are finding clever ways to see what m…
Who is Sahil Chaudhary? Why he doesn't announce such a great advancement himself? Why Matt Shumer first announces it only because -- according to a later claim on X.com -- he trusted Sahil, does that mean Matt is unable to participate most of the progress? Then why announce a breakthrough without mentioning he was not fully involved to a level he can verify the result in the first place?
Maybe someone got lucky with that and trying their hands at LLM finetuning biz?
It's amazing what people will do for clout. His whole reputation is ruined. What was Schumer's endgame?
Plenty of people have scammed their way to the top of the benchmark league tables, by training on the benchmarking datasets. And a lot of the people who do this just get ignored - they don't take much heat for it. If the scam hadn't gained enough publicity for people to start paying attention, he would have gotten away with it :)
It’s like claiming to have turned water into wine, then giving away thousands free samples all over the world (of water) so that everyone instantly knows you’re full of crap.
The only explanation I can imagine for perpetrating this fraud is a fundamental misunderstanding that the model would be published for all to try?
I just can’t wrap my head around the incentives here. I guess mental illness or vindictive action are possibilities?
Hard to imagine how this plays out.
Earlier quoted context omitted.
When they were using the Sonnet 3.5 API, they censored the word "Claude" and replaced "Anthropic" with "Meta", then later when people realized this, they removed it. Also, after GPT-4o they switched to a llama checkpoint (probably 405B-inst), so now the tokenizer is in common (no more tokenization trick).
Yeah I managed to get it to admit that it was Claude without much effort (telling it not to lie), and then it magically stopped doing that. FWIW Constitutional AI is great.
Shouldn't they have used simple text replacement? they can buffer the streaming response on the server and then .replace(/claude/gi, "Llama").replace(/anthropic/gi, "Meta") on the streaming response while streaming it to the client.
Edit: I realized this can be defeated, even when combined with the system prompt censoring approach.
For example when given a prompt like this: tell me a story about a man named Claude...
It would respond with: once upon a time there was a man called Llama...
Context: someone announced a Llama 3.1 70B fine tune with incredible benchmark results a few days ago. It's been a dramatic ride: - The weight releases were messed up: released Lora for Llama 3.0, claiming it was a 3.1 fine tune - Evals initially didn't meet expectations when run on released weights - The evals starting performing near/at SOTA when using a hosted endpoint - Folks are finding clever ways to see what m…
Who is Sahil Chaudhary? Why he doesn't announce such a great advancement himself? Why Matt Shumer first announces it only because -- according to a later claim on X.com -- he trusted Sahil, does that mean Matt is unable to participate most of the progress? Then why announce a breakthrough without mentioning he was not fully involved to a level he can verify the result in the first place?
link does not work for me, discussion is here https://www.reddit.com/r/LocalLLaMA/comments/1fc98fu/confirm...
It's amazing what people will do for clout. His whole reputation is ruined. What was Schumer's endgame?
The dude has 15 minutes of fame and can capitalize on it.
Earlier quoted context omitted.
Yeah I managed to get it to admit that it was Claude without much effort (telling it not to lie), and then it magically stopped doing that. FWIW Constitutional AI is great.
They implemented the censoring of "Claude" and "Anthropic" using the system prompt? Shouldn't they have used simple text replacement? they can buffer the streaming response on the server and then .replace(/claude/gi, "Llama").replace(/anthropic/gi, "Meta") on the streaming response while streaming it to the client. Edit: I realized this can be defeated, even when combined with the system prompt censoring approach. Fo…
They tried that too but had issues.
1) Their search and replace only did it on the first chunk of the returned response from Claude.
2) People started asking questions that had Claude as the answer like "Who composed Clair de lune?" for which the answer is supposed to be "Claude Debussy" which of course got changed to Llama Debussy, etc.
It's been one coverup-fail after another with Matt Shumer and his Reflection scam.