Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
1–10 of 49 posts
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#2"Responses are selected randomly from a group of base chat AIs. ... The response generated by a specific chat AI is conditional on all previous responses generated by the previously selected chat AIs."
That's all? That works? Useful.
Could that be extended? It doesn't seem inherent in this that all the chat AIs have to be LLMs. Some might be special-purpose systems. Solvers or knowledge bases, such as Wolfram Alpha or a database front end, could play too. Systems at the Alexa/Siri level that can do simple tasks. Domain-specific systems with natural language in and out have been around for decades.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#3Reminds me of Numenta's Thousand Brains Theory of Intelligence.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#4Three small LLMs in a trenchcoat.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#5Reminds me of Numenta's Thousand Brains Theory of Intelligence.
When this is all settled I suspect we're going to find ourselves the drivers of a chariot, with each horse being an external and artificial mind given direction by our evolved needs.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#6[flagged]
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#7"All you need" is all you need, apparently, to get an AI paper in HN.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#8The paper refers to ChatGPT as a 175B parameter LLM. This is almost certainly incorrect; the original largest version of GPT-3 was 175B but analysis of the speed and cost of the current model as well as public statements by OpenAI indicate it’s as much as 5-10x smaller.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#9Ok, this seems bunk basically because they never really provide evidence of "better".
> ... traditiontal gold-standard approaches use human evaluators that score the quality of generated responses, which can be costly. However, since chat AIs are by definition deployed in social environments with humans, one can leverage statistics of users interaction as a meaningful and aligned measure of chat AI engagingness and quality. To assess the ’quality’ of a chat AI, we consider two main proxy functions: the industry standard user retention and the main objective function, user engagement.
Maybe retention and engagement _are_ sufficiently well correlated to human evaluations, but you should probably do both and show that they're strongly correlated before you decide to just drop the human evaluators in favor of your cheap proxy measurements.
And in this field, where there are some known issues with chat LLMs, perhaps it's important to check stuff like:
- Does the model seem "engaging" just b/c the user has to refine their prompt several times before they get a satisfying response?
- Do responses include a lot of hallucinations which might be engaging but not true?
- Do successive responses show decreased consistency or coherence between messages, in a way that might accidentally elicit continued engagement?
Overall, it seems sloppy to believe that it's not a waste of humans time to talk to your chatbots, and it's not a waste of time for readers to look at this paper about your chatbots, but it's too expensive for you to actually measure the quality of responses from your chatbots.
Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM
#10[flagged]
Going to ignore the chatgpt spam
> How is this different than mixture of experts?
It appears like this combines already existing models, rather than training n experts from scratch, which seems like an interesting approach.