Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if we remove 50 percent of Llama?

neuralmagic.com

Post reply on HN
Print view View original

131–139 of 139 posts

Re: What happens if we remove 50 percent of Llama?

#131

Post by ben_w »

spiritplumber wrote: post #121
ben_w wrote: post #56

Earlier quoted context omitted.

One of the ways I think all this will plausibly go wrong is, as per the fictional Solarians, one group of humans that have AI that are trained to only recognise that group as being real humans.

Stupidly, this was a problem a few years ago when face recognition would only pick up whites and asians.

Wow, somehow I forgot about that despite having read about it multiple times.

I was only thinking just about malicious de-humanising, but yes, you're right, that absolutely is a valid example.

Re: What happens if we remove 50 percent of Llama?

#132

Post by ljlolel »

regularfry wrote: post #127
qeternity wrote: post #39

Earlier quoted context omitted.

It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token).

Weirdly it does have to do with words, but not intentionally. Mechanically the routing is per-token, but the routing is frequently stable across a word as an emergent property. At least, that's how I read the mixtral paper.

Yep.

Also note that I’m ELI5 so saying word is fine.

Re: What happens if we remove 50 percent of Llama?

#133

Post by incrudible »

quotemstr wrote: post #79
Workaccount2 wrote: post #74

Earlier quoted context omitted.

>With the discovery that transformers lack reasoning capabilities The only paper I have seen claiming this studied only lightweight open-source models (<27B, mostly 2B and 8B). The also included o1 and 4o for reference, which kind of broke their hypothesis, but they just left that part out of the conclusion. Not even kidding, their graphs show o1 and 4o having strong performance in their benchmarks, but the conclusio…

A vocal minority of researchers are essentially human chauvinists --- they "want to believe" that LLMs can't "really" perform this or that part of cognition even though the evidence is blinding that they can. (Anyone who genuinely believes that LLMs can't reason at all has never used an LLM.) These researchers start with their conclusion and work backwards to an argument, making their work seductive but useless.

The problem is in being able to discern reasoning from patterns that happen to exist in the training data. There are plenty of tricks you can play on an LLM by subverting the expectations it must necessarily have due to its training data. A human might fall into the same trap, but can then reason themselves out of it, whereas an LLM tends to double down on its mistake.

Re: What happens if we remove 50 percent of Llama?

#134

Post by visarga »

mewpmewp2 wrote: post #118
visarga wrote: post #107

Earlier quoted context omitted.

The fact that you can prune a model will not make it smarter, the wall still stands. I think what explains the wall is the fact that we can't scale organic data exponentially, and we have already covered the most useful types. Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data on…

Don't we still have a lot of video, and other non text real world data to go with? Feels like a possible potential break from there.

Generally speaking text only models manage to learn a huge amount about the visual world. So when you put the model train on video it might have less to learn. Video is also less abstract than text, generally. But I am sure we can still extract useful learning from videos, it's probably expensive, but we'll have to do that at some point.

Re: What happens if we remove 50 percent of Llama?

#136

Post by griomnib »

visarga wrote: post #107
celltalk wrote: post #46

All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…

The fact that you can prune a model will not make it smarter, the wall still stands. I think what explains the wall is the fact that we can't scale organic data exponentially, and we have already covered the most useful types. Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data on…

Given how much of the web is ai generated slop now, I think going forward it’s even worse than you suggest.

I have a copy of refined web locally so I have a billion pre-chatgpt documents for my long term use.

Re: What happens if we remove 50 percent of Llama?

#137

Post by fasa99 »

DHRicoF wrote: post #87
bravura wrote: post #84

Earlier quoted context omitted.

I seem to recall that there a recent theory paper that got a best paper award, but can't find it. If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space. [This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.] So wi…

I don't see that contuer-intuitive at all. If you have a barrier in your cost function in 1d model you have to cross over it no matter what. In 2d it could be only a mount that you can go around. More dimensions mean more ways to go around.

This is also how the human brain works. A young babby will have something more similar to a fully connected network. Versus a Biden type elderly brain will be more of a sparse minimally connected feed forward net. The question is (1) can this be adjusted dynamically in silico and (2) if we succeed in that, does fine-tuning still work?

Re: What happens if we remove 50 percent of Llama?

#138

Post by int_19h »

bombela wrote: post #102
int_19h wrote: post #95

Earlier quoted context omitted.

There is some improvement going from 4-bit to 8-bit quantization, but if you have VRAM to spare for that, you usually see more benefit from running a 2x larger model at 4-bit. So in scenarios where an LM already fits the existing VRAM budget, I would expect larger models instead. The other thing is that VRAM is used not just for the weights, but also for prompt processing, and this last part grows proportionally as y…

For readability, I recommend reserving "b" for bits, "B" for byte, "p" for parameter. I assume in your post that "30b" meant 30 billion, or in other words, 30Gp (giga-parameter). Furthermore is 24Gb of VRAM 24 gigabits (power of 10), or 24 gibibits (power of 2)?

For readability I'm using the same convention that is generally used for these applications, where if you see "-Nb" after a model name, it always refers to the number of parameters. I have never once seen "p" for "parameter", never mind terms like "giga-parameter". Most certainly if you go searching for models on HuggingFace etc, you'll have to deal with "30b" etc terminology whether you like it or not.

With VRAM, this quite obviously refers to the actual amount that high-end GPUs have, and I even specifically listed which ones I have in mind, so you can just look up their specs if you genuinely don't know the meaning in this context.

Re: What happens if we remove 50 percent of Llama?

#139

Post by scotty79 »

fasa99 wrote: post #137
DHRicoF wrote: post #87

Earlier quoted context omitted.

I don't see that contuer-intuitive at all. If you have a barrier in your cost function in 1d model you have to cross over it no matter what. In 2d it could be only a mount that you can go around. More dimensions mean more ways to go around.

This is also how the human brain works. A young babby will have something more similar to a fully connected network. Versus a Biden type elderly brain will be more of a sparse minimally connected feed forward net. The question is (1) can this be adjusted dynamically in silico and (2) if we succeed in that, does fine-tuning still work?

You don't have to compare to old age. Even 10 year old child has its brain pruned immensely when compared to its babyself.
Post reply on HN