How does the blending work? I'm imagining installing a bunch of "AIs" and having them all work together intelligently.
Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
41–49 of 49 posts
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#42I've said this a few times previously yet I certainly want to say it again - "All You Need" titles are definetely not what all we need.
I'm working on a paper entitled "All You Need Considered Harmful" to address this over-usage of the phrase.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#43Earlier quoted context omitted.
I think it was leaked that it is 20B now.
It was mentioned to be a 20B in a comparison table in a paper co-written by Microsoft, but they've since claimed that it's just an error, and I mean, they'd need to be sitting on some really impressive distilling techniques to shrink a 175B model down to 20B with only a slight drop in performance.
OpenAI have been sitting on GPT4 for months, and on the basemodel even longer. I would not be surprised if they did some or all of the distillation of the model with GPT4.
Mixtral is 56B combined, if we subtract a little for MoE inefficiencies we could say that Mixtral is about 40B combined. This is a 2x increase over 20B. We have seen new models beat others twice the size.
That and a massive amount of excellent data for alignment should produce some great results.
I don't think it's out of the realm of possibility that 20B is real.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#44I've said this a few times previously yet I certainly want to say it again - "All You Need" titles are definetely not what all we need.
"All you need are better titles!"
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#45Now that I think about it, doesn't this "technique" triple the amount of compute and memory per generated token since each model needs to also compute and store the KV values for the two previous tokens it didn't generate and thus has never seen? Edit: On second thought, depending on how it's actually implemented the other two tokens are probably ran through the model in parallel so it shouldn't be all that much slow…
It doesn’t generate three responses for every turn. It randomly picks a model for every response, the claim being that the switching between different models leads to better conversations because of the diversity of each model’s training.
Correct me if I'm wrong but usually when you do normal token by token inference in a transformer you store calculations made in the previous step in a KV cache so you can reuse it instead of calculating it all over again.
But here since the previous few tokens were produced by another model, the current model has never seen them and as such, by definition, doesn't have those calculations stored, but it still needs them to properly calculate attention for the new token.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#46How does the blending work? I'm imagining installing a bunch of "AIs" and having them all work together intelligently.
According to the paper, it just selects one AI randomly and then uses it to generate the next response.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#47I really would like them to compare to Gpt4 instead of claiming victory when matching 3.5. To me GPT4 is the first usable one for a lot of professional uses. 3.5 is fun and gets some stuff right but it’s like a demo.
Honestly, the baseline models they test and blend are really terrible as well. Especially Pygmalion 6B, which is like ancient history. A Yi 34B or Mixtral finetune on the same data would blow them out of the water. Probably blow ChatGPT 3.5 out of the water as well.
>the baseline models they test and blend are really terrible as well.
If the effect is there I would guess a few bad models should outperform a mediocre one, and a few mediocre ones should outperform a state-of-the-art one.
Of course it would be good to show the same again with GPT4 and maybe 3 GPT3.5 size models, but it's not necessary to show that such an effect exists, and maybe cost prohibitive for them as a research team. Now whether their methodology for proving this effect is correct is another discussion.
Personally I don't find these results surprising, our brain is also somewhat compartmentalized, why wouldn't the same hold for a good AI system?
The more difficult part is, how do you train these subnetworks optimally.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#48Earlier quoted context omitted.
It doesn’t generate three responses for every turn. It randomly picks a model for every response, the claim being that the switching between different models leads to better conversations because of the diversity of each model’s training.
Correct me if I'm wrong but usually when you do normal token by token inference in a transformer you store calculations made in the previous step in a KV cache so you can reuse it instead of calculating it all over again. But here since the previous few tokens were produced by another model, the current model has never seen them and as such, by definition, doesn't have those calculations stored, but it still needs th…
It doesn’t appear to be token-by-token inference. Each new completion uses a different model, but the new completion is entirely created by that model.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#49"Responses are selected randomly from a group of base chat AIs. ... The response generated by a specific chat AI is conditional on all previous responses generated by the previously selected chat AIs." That's all? That works? Useful. Could that be extended? It doesn't seem inherent in this that all the chat AIs have to be LLMs. Some might be special-purpose systems. Solvers or knowledge bases, such as Wolfram Alpha o…
Seems a lot like ensembling methods for traditional predictive models.
Not quite. The outputs of the models become part of the prompt history of all the models. So they can assist each other. For example, one model might produce a highly technical reply. Then the user can ask for explanations, to be provided by a model with better language capability but less domain expertise.