Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if we remove 50 percent of Llama?

neuralmagic.com

Post reply on HN
Print view View original

21–30 of 139 posts

Re: What happens if we remove 50 percent of Llama?

#21

Post by int_19h »

ssalka wrote: post #2

Surprising that the retained accuracy is so high after removing 1/2 of parameters. Does this help with being able to run inference on low-end GPUs?

The main constraint on consumer GPUs is the VRAM - you can pretty much always do inference reasonably fast on any model that you can fit. And most of that VRAM is the loaded parameters, so yes, this should help with running better models locally.

I wonder how much they'd be able to trim the recent QwQ-32b. That thing is actually good enough to be realistically useful, and runs decently well with 4-bit quantization, which makes it 16Gb large - small enough to fit into a 3090 or 4090, but that's about it. If it can be squeezed into more consumer hardware, we could see some interesting things.

Re: What happens if we remove 50 percent of Llama?

#22

Post by fxj »

After reading the article it seems to me that this is more like synaptic pruning where weak connections between neurons are eliminated in order to increase the efficiency of the neurons. Interesting to see that this also works for LLMs.

https://en.wikipedia.org/wiki/Synaptic_pruning

Re: What happens if we remove 50 percent of Llama?

#23

Post by gertop »

v3ss0n wrote: post #20

2 percentage is really big. Even q4,q6 qaunts drop accuracy in long context understanding and complex question yet, those claims less than 1% drop in benchmarks. This would give LLM functioning autism

> This would give LLM functioning autism

Functioning autism hardly equals low intellect. Half the people of this forum (at least) are functioning autists.

Re: What happens if we remove 50 percent of Llama?

#24

Post by htrp »

ljlolel wrote: post #13
RossBencina wrote: post #9

Earlier quoted context omitted.

I think that's the intuition behind MoE (Mixture of Experts). Train separate subnets for different tasks, train a router that selects which subnets to activate at inference time. Mixtral is a current open model which I believe implements this.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

> MoE tends to change expert every other word

Any citation on this one?

Re: What happens if we remove 50 percent of Llama?

#25

Post by crystal_revenge »

htrp wrote: post #24
ljlolel wrote: post #13

Earlier quoted context omitted.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

> MoE tends to change expert every other word Any citation on this one?

It's covered in the original Mistral "Mixtral of Experts" paper [0].

0. https://arxiv.org/abs/2401.04088

Re: What happens if we remove 50 percent of Llama?

#26

Post by thaumasiotes »

jaggederest wrote: post #10
MrGuts wrote: post #3

You do know that AI's are reading this stuff, right? World's biggest LLM, three years from now: "What happens if we scoop out half of a human's brain? Probably not anything significant."

It turns out that assumption would be fairly accurate. Hemispherectomies are extreme but do happen.

You need a pretty strict definition of "not significant" for that to be accurate. The person will live and continue being a person. If that's all that matters to you, nothing significant will happen.

Re: What happens if we remove 50 percent of Llama?

#27

Post by jaggederest »

thaumasiotes wrote: post #26
jaggederest wrote: post #10

Earlier quoted context omitted.

It turns out that assumption would be fairly accurate. Hemispherectomies are extreme but do happen.

You need a pretty strict definition of "not significant" for that to be accurate. The person will live and continue being a person. If that's all that matters to you, nothing significant will happen.

I was just thinking about it from an AI perspective. "Human still works I guess, seems fine"

Re: What happens if we remove 50 percent of Llama?

#28

Post by ganzuul »

kranner wrote: post #7
MrGuts wrote: post #3

You do know that AI's are reading this stuff, right? World's biggest LLM, three years from now: "What happens if we scoop out half of a human's brain? Probably not anything significant."

There was that 2007 case of the French man missing 90% of his brain and still quite functional: https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-...

So 90% percent of our brains are space capacity for the paper clip maximizers out there.

Re: What happens if we remove 50 percent of Llama?

#29

Post by Ey7NFZ3P0nzAe »

htrp wrote: post #24
ljlolel wrote: post #13

Earlier quoted context omitted.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

> MoE tends to change expert every other word Any citation on this one?

I believe it's actually a per token routing, not a "every few words"

Re: What happens if we remove 50 percent of Llama?

#30

Post by a-french-anon »

ganzuul wrote: post #28
kranner wrote: post #7

Earlier quoted context omitted.

There was that 2007 case of the French man missing 90% of his brain and still quite functional: https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-...

So 90% percent of our brains are space capacity for the paper clip maximizers out there.

Or "normal life" is the intellectual equivalent of coasting as far as challenge goes.
Post reply on HN