The False Promise of Imitating Proprietary LLMs
1–10 of 90 posts
Re: The False Promise of Imitating Proprietary LLMs
#2The authors conduct automated, more methodical evaluations of LLMs finetuned to imitate ChatGPT outputs, and find that, despite superficial/informal appearances to the contrary, the base LLMs close little to none of the gap to ChatGPT on tasks that are not heavily supported in the imitation data.
It's not good news for the open LLM ecosystem.
Re: The False Promise of Imitating Proprietary LLMs
#3The authors conduct automated, more methodical evaluations of LLMs finetuned to imitate ChatGPT outputs, and find that, despite superficial/informal appearances to the contrary, the base LLMs close little to none of the gap to ChatGPT on tasks that are not heavily supported in the imitation data. It's not good news for the open LLM ecosystem.
I don't know if it's bad news per say. It helps to know where to deploy a tool, it's limitations and where to focus to build something competitive / better.
Re: The False Promise of Imitating Proprietary LLMs
#4This is an important study and I've been waiting for something like this ever since Alpaca and the following wave of imitating models that have had lackluster, non rigorous evaluation.
Re: The False Promise of Imitating Proprietary LLMs
#5This isn't a new result really. We already know through the gpt-4 paper that rlhf style fine-tuning just makes the model more compliant, not more capable.
Re: The False Promise of Imitating Proprietary LLMs
#6This isn't a new result really. We already know through the gpt-4 paper that rlhf style fine-tuning just makes the model more compliant, not more capable.
Exactly. Few hundreds of thousands of interactions with chatgpt is definitely too less for model to learn lot of new things. The thing it does well is make them much better at following instructions. It also makes it much better at working with given context.
Re: The False Promise of Imitating Proprietary LLMs
#7If they really didn't test anything bigger than 13b, as their abstract states, then this doesn't even seem worth reading through.
Re: The False Promise of Imitating Proprietary LLMs
#8nobody with lots of experience with proprietary LMs is surprised
Re: The False Promise of Imitating Proprietary LLMs
#9nobody with lots of experience with proprietary LMs is surprised
And neither is anyone who has played with these new LLMs, found them so-so, and wondered whether the hype was warranted.
Re: The False Promise of Imitating Proprietary LLMs
#10If they really didn't test anything bigger than 13b, as their abstract states, then this doesn't even seem worth reading through.
my eyebrows went up at a number of choices made in their assessment