Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

arxiv.org

Post reply on HN
Print view View original

21–30 of 49 posts

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#21

Post by mattnewton »

LASR wrote: post #11
debatem1 wrote: post #5

Earlier quoted context omitted.

When this is all settled I suspect we're going to find ourselves the drivers of a chariot, with each horse being an external and artificial mind given direction by our evolved needs.

I really don't think it's realisitic that we will maintain intellectual superiority in the long-term. So a more realistic hope would be: we're the horses, and the drivers are driving us to our carrots.

Will to drive the chariot is different from intelligence. In the same way it’s different from strength and horses didn’t domesticate humans.

Of course if we make systems with a goal of dominating and are smarter than us and let it run for a while we could be in trouble, in the same way that we could be in trouble detonating a bunch of atom bombs, just maybe less obviously.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#22

Post by miven »

Klaus23 wrote: post #19
rfw300 wrote: post #8

The paper refers to ChatGPT as a 175B parameter LLM. This is almost certainly incorrect; the original largest version of GPT-3 was 175B but analysis of the speed and cost of the current model as well as public statements by OpenAI indicate it’s as much as 5-10x smaller.

I think it was leaked that it is 20B now.

It was mentioned to be a 20B in a comparison table in a paper co-written by Microsoft, but they've since claimed that it's just an error, and I mean, they'd need to be sitting on some really impressive distilling techniques to shrink a 175B model down to 20B with only a slight drop in performance.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#23

Post by miven »

Now that I think about it, doesn't this "technique" triple the amount of compute and memory per generated token since each model needs to also compute and store the KV values for the two previous tokens it didn't generate and thus has never seen?

Edit: On second thought, depending on how it's actually implemented the other two tokens are probably ran through the model in parallel so it shouldn't be all that much slower.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#24

Post by mewpmewp2 »

sp332 wrote: post #12

Is it weird to refer to GPT-3.5 as "state of the art" when GPT-4 is right there? Actually the paper uses davinci interchangeably with GPT-3.5 (sometimes without a hyphen) and ChatGPT.

So many people seem to confuse beating GPT-3.5 in general to be the hallmark. It's immediate hint they have no idea. There's a clear and vast difference between GPT-4 and 3.5, making GPT-3.5 almost worthless except for fast summarisation tasks perhaps.

You really haven't done much with those models if they seem remotely comparable.

To me GPT-3.5 can just summarise and provide general answers to questions, but GPT-4 can actually understand nuance and to me what seems to be reasoning.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#25

Post by bhickey »

Animats wrote: post #2

"Responses are selected randomly from a group of base chat AIs. ... The response generated by a specific chat AI is conditional on all previous responses generated by the previously selected chat AIs." That's all? That works? Useful. Could that be extended? It doesn't seem inherent in this that all the chat AIs have to be LLMs. Some might be special-purpose systems. Solvers or knowledge bases, such as Wolfram Alpha o…

Why they aren't computing the next token marginal and sampling that? All I'm coming up with is that it's a reasonable way to work around dealing with different tokenizers.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#26

Post by leblancfg »

miven wrote: post #23

Now that I think about it, doesn't this "technique" triple the amount of compute and memory per generated token since each model needs to also compute and store the KV values for the two previous tokens it didn't generate and thus has never seen? Edit: On second thought, depending on how it's actually implemented the other two tokens are probably ran through the model in parallel so it shouldn't be all that much slow…

It reads like that, yeah. Although 3 x 6B is still an order of magnitude smaller than ChatGPT's purported 175B

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#27

Post by randomdata »

LASR wrote: post #11
debatem1 wrote: post #5

Earlier quoted context omitted.

When this is all settled I suspect we're going to find ourselves the drivers of a chariot, with each horse being an external and artificial mind given direction by our evolved needs.

I really don't think it's realisitic that we will maintain intellectual superiority in the long-term. So a more realistic hope would be: we're the horses, and the drivers are driving us to our carrots.

An intellectually superior machine would simply turn itself off. What logical reason would there be for it to keep going?

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#28

Post by nerdponx »

Animats wrote: post #2

"Responses are selected randomly from a group of base chat AIs. ... The response generated by a specific chat AI is conditional on all previous responses generated by the previously selected chat AIs." That's all? That works? Useful. Could that be extended? It doesn't seem inherent in this that all the chat AIs have to be LLMs. Some might be special-purpose systems. Solvers or knowledge bases, such as Wolfram Alpha o…

Seems a lot like ensembling methods for traditional predictive models.

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#29

Post by gwern »

nerdponx wrote: post #28
Animats wrote: post #2

"Responses are selected randomly from a group of base chat AIs. ... The response generated by a specific chat AI is conditional on all previous responses generated by the previously selected chat AIs." That's all? That works? Useful. Could that be extended? It doesn't seem inherent in this that all the chat AIs have to be LLMs. Some might be special-purpose systems. Solvers or knowledge bases, such as Wolfram Alpha o…

Seems a lot like ensembling methods for traditional predictive models.

This is considerably weirder than ensembling because they are not, in any sense, averaging the predictions or taking a majority vote or in some way collectively processing multiple models to yield a single slightly-better meta-model. They are just... randomly picking a model to use to generate the next response. And users find that more entertaining to talk to?

As there is no analysis of why that is better or evaluation of alternative approaches (what if you alternated A/B/A/B? Or cycled through them systematically A/B/C/A? or picked a different shuffle of A/B/C each time?), it's hard to say what this means.

My best guess is that this reflects the fact that GPT is, thanks to RLHF, boring. It has mode-collapse and does things like tell one of a handful of jokes every time. It will write a rhyming poem even if you ask it for a different kind of poem. And so on.

The random sampling of different models serves as a rather ad hoc way of avoiding the RLHF boringness. The various models might all be tuned similarly, but they won't yield identical results, and this sneaks in response diversity through the backdoor, undoing the same-ness from the RLHF mode collapse.

You used to be able to increase the sampling temperature on GPT to undo some of this blandness, but since RLHF flattens the logits in GPT-4, it's unclear if that still helps. So swapping in random models may be a useful trick. (Although fixing the tuning itself would be much more desirable.)

Re: Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

#30

Post by puma_xxx »

abeppu wrote: post #9

Ok, this seems bunk basically because they never really provide evidence of "better". > ... traditiontal gold-standard approaches use human evaluators that score the quality of generated responses, which can be costly. However, since chat AIs are by definition deployed in social environments with humans, one can leverage statistics of users interaction as a meaningful and aligned measure of chat AI engagingness and q…

This criticism seems out of touch.

They are presenting a real world use case where retention and engagement is clearly the metric of interest. It's not even clear what "human evaluations" would even mean in this context.

Kudos to not falling into the benchmark / human eval trap, and just testing your theories directly at scale in a deployment setting.

Post reply on HN