Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if we remove 50 percent of Llama?

neuralmagic.com

Post reply on HN
Print view View original

81–90 of 139 posts

Re: What happens if we remove 50 percent of Llama?

#81

Post by elcomet »

devsda wrote: post #6

I don't understand LLMs enough to know if this is a silly question or not. Is it possible to build domain specific smaller models and merge/combine them at query/run time to give better response or performance instead of one large all knowing model that learns everything ?

It's possible, the question is how to choose which submodel will be used for a given query.

You can use a specific LLM, or a general larger LLM to do this routing.

Also, some work suggest using smaller llms to generate multiple responses and use a stronger and larger model to rank the responses (which is much more efficient than generating them)

Re: What happens if we remove 50 percent of Llama?

#82

Post by jdietrich »

zamalek wrote: post #76
Workaccount2 wrote: post #74

Earlier quoted context omitted.

>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…

https://arxiv.org/abs/2410.05229 An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm. Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.

I just don't see how anyone can see a study comparing the reasoning abilities of various LLMs, see that large LLMs have better reasoning abilities and conclude that LLMs can't reason. LLMs don't have human-like reasoning abilities, but it's just obviously true that they have some capacity for reasoning; that ability seems to scale roughly linearly with model size and training FLOPs.

Re: What happens if we remove 50 percent of Llama?

#83

Post by xpuente »

zug_zug wrote: post #64
xpuente wrote: post #63

Earlier quoted context omitted.

The issue is that no one fully understands why synaptic pruning occurs in biology. Large language models have no direct connection to biological systems, and pruning in LLMs is no exception.

Really? It seems obvious to me. During the learning stage we want input from every variable so that we are sure that we don't omit a variable that turns out to be essential for the calculation. However in any calculation a human does 99.9999% of variables are irrelevant (e.g. what day of the week it is, am I sleepy, etc), so of course the brain wouldn't use resources to keep connections that aren't relevant to a give…

As far as I know, pruning is related to age. At birth, we have a massive number of silent synapses. As we grow older, those that remain unused (i.e., inactive) tend to disappear. This process involves a delicate mechanism, including components of the immune system.

The unfortunate reality is that no one truly understands how memory works. Many theories are floating around, but the fundamental components remain elusive. One thing is certain: it is quite different from backpropagation. Thankfully, our brains do not suffer from catastrophic forgetting.

Re: What happens if we remove 50 percent of Llama?

#84

Post by bravura »

celltalk wrote: post #46

All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…

I seem to recall that there a recent theory paper that got a best paper award, but can't find it.

If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space.

[This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.]

So with this hand-wavey argument, it might be the case that two-phase training is needed: A large overcomplete pretraining focused on assimilating all the knowledge, and a second that makes it compact. Other, that there is a hyperparameter that controls overcompleteness vs compactness and you adjust it over training.

Re: What happens if we remove 50 percent of Llama?

#85

Post by moralestapia »

jdietrich wrote: post #82
zamalek wrote: post #76

Earlier quoted context omitted.

https://arxiv.org/abs/2410.05229 An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm. Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.

I just don't see how anyone can see a study comparing the reasoning abilities of various LLMs, see that large LLMs have better reasoning abilities and conclude that LLMs can't reason. LLMs don't have human-like reasoning abilities, but it's just obviously true that they have some capacity for reasoning; that ability seems to scale roughly linearly with model size and training FLOPs.

Yes, but is human-reasoning on the same spectrum as LLM-reasoning? Meaning that only scale will turn the latter into the former?

No definitive answer yet, but my bet is on no.

Re: What happens if we remove 50 percent of Llama?

#86

Post by Scene_Cast2 »

bravura wrote: post #84
celltalk wrote: post #46

All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…

I seem to recall that there a recent theory paper that got a best paper award, but can't find it. If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space. [This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.] So wi…

The lottery ticket hypothesis paper from 2018?

Re: What happens if we remove 50 percent of Llama?

#87

Post by DHRicoF »

bravura wrote: post #84
celltalk wrote: post #46

All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…

I seem to recall that there a recent theory paper that got a best paper award, but can't find it. If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space. [This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.] So wi…

I don't see that contuer-intuitive at all. If you have a barrier in your cost function in 1d model you have to cross over it no matter what. In 2d it could be only a mount that you can go around. More dimensions mean more ways to go around.

Re: What happens if we remove 50 percent of Llama?

#88

Post by Nevermark »

moralestapia wrote: post #85
jdietrich wrote: post #82

Earlier quoted context omitted.

I just don't see how anyone can see a study comparing the reasoning abilities of various LLMs, see that large LLMs have better reasoning abilities and conclude that LLMs can't reason. LLMs don't have human-like reasoning abilities, but it's just obviously true that they have some capacity for reasoning; that ability seems to scale roughly linearly with model size and training FLOPs.

Yes, but is human-reasoning on the same spectrum as LLM-reasoning? Meaning that only scale will turn the latter into the former? No definitive answer yet, but my bet is on no.

Agreed, and I think the answer is pretty clear.

Large models successful now have dodged recurrent architecture, which is harder to train but allows for open ended inference steps, which would allow straightforward scaling to any number of reasoning steps.

At some point, recurrent connections are going to get re-incorporated into these models.

Maybe two stage training. First stage, learn to integrate as much information as well as possible, without recurrence. As is happening now. Second training stage, embed that model in a larger iterative model, and train for variable step reasoning.

Finally, successful iterative reasoning responses can be used as further examples for the non-iterative module.

This would be similar to how we reason in steps at first, in unfamiliar areas. But quickly learn to reason with faster direct responses, as we gain familiarity.

We continually fine tune our fast mode on our own more powerful slow mode successes.

Re: What happens if we remove 50 percent of Llama?

#89

Post by wat10000 »

zamalek wrote: post #76
Workaccount2 wrote: post #74

Earlier quoted context omitted.

>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…

https://arxiv.org/abs/2410.05229 An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm. Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.

Has anyone done this sort of test on people?
Post reply on HN