Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if we remove 50 percent of Llama?

neuralmagic.com

Post reply on HN
Print view View original

31–40 of 139 posts

Re: What happens if we remove 50 percent of Llama?

#31

Post by xvector »

j16sdiz wrote: post #19
ljlolel wrote: post #13

Earlier quoted context omitted.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

> Nobody understands how or why the router chooses the expert. It’s so early. Nobody understand how LLM works either. Is LLM as "early" as MoE ?

LLMs are really well understood, what do you mean? You can see the precise activations and token probabilities for every next token. You can abliterate the network however you'd like to suppress or excite concepts of your choosing.

Re: What happens if we remove 50 percent of Llama?

#32

Post by xvector »

kranner wrote: post #7
MrGuts wrote: post #3

You do know that AI's are reading this stuff, right? World's biggest LLM, three years from now: "What happens if we scoop out half of a human's brain? Probably not anything significant."

There was that 2007 case of the French man missing 90% of his brain and still quite functional: https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-...

This is really interesting from the perspective of gradual replacement/mind uploading: what is the absolute minimum portion of the brain that we would have to target?

Understanding this could probably make the problem easier by some factor (but not "easy" in any sense.)

Re: What happens if we remove 50 percent of Llama?

#33

Post by slaucon »

> “By sourcing and filtering only the highest-quality and most representative data for LLM use cases, we reduced the pretraining set to just 13 billion tokens—drastically cutting the environmental impact of further training while preserving performance.”

Would love to know more about how they filtered the training set down here and what heuristics were involved.

I think that the models we use now are enormous for the use cases we’re using them for. Work like this and model distillation in general is fantastic and sorely needed, both to broaden price accessibility and to decrease resource usage.

I’m sure frontier models will only get bigger, but I’d be shocked if we keep using the largest models in production for almost any use case.

Re: What happens if we remove 50 percent of Llama?

#35

Post by concerndc1tizen »

int_19h wrote: post #21
ssalka wrote: post #2

Surprising that the retained accuracy is so high after removing 1/2 of parameters. Does this help with being able to run inference on low-end GPUs?

The main constraint on consumer GPUs is the VRAM - you can pretty much always do inference reasonably fast on any model that you can fit. And most of that VRAM is the loaded parameters, so yes, this should help with running better models locally. I wonder how much they'd be able to trim the recent QwQ-32b. That thing is actually good enough to be realistically useful, and runs decently well with 4-bit quantization, w…

Does this mean that the model will be half the size?

If a 32B model@4bit normally requires 16 GB VRAM, at half the size, it could be run @8bit with 16 GB VRAM?

Isn't that tradeoff a great improvement? I assume the improved bit precision will more than compensate for the loss related to removal?

Re: What happens if we remove 50 percent of Llama?

#36

Post by SubiculumCode »

gertop wrote: post #23
v3ss0n wrote: post #20

2 percentage is really big. Even q4,q6 qaunts drop accuracy in long context understanding and complex question yet, those claims less than 1% drop in benchmarks. This would give LLM functioning autism

> This would give LLM functioning autism Functioning autism hardly equals low intellect. Half the people of this forum (at least) are functioning autists.

No, but it's also true that almost 40% of autists have intellectual disabilities: https://www.cdc.gov/mmwr/volumes/72/ss/ss7202a1.htm

That said, the parent comment is just silly and wrong.

Re: What happens if we remove 50 percent of Llama?

#38

Post by qeternity »

RossBencina wrote: post #9
devsda wrote: post #6

I don't understand LLMs enough to know if this is a silly question or not. Is it possible to build domain specific smaller models and merge/combine them at query/run time to give better response or performance instead of one large all knowing model that learns everything ?

I think that's the intuition behind MoE (Mixture of Experts). Train separate subnets for different tasks, train a router that selects which subnets to activate at inference time. Mixtral is a current open model which I believe implements this.

This is not how MoEs work at all. They are all trained together, often you have multiple experts activated for a single token. They are not domain specific in any way that is understandable by humans.

Re: What happens if we remove 50 percent of Llama?

#39

Post by qeternity »

ljlolel wrote: post #13
RossBencina wrote: post #9

Earlier quoted context omitted.

I think that's the intuition behind MoE (Mixture of Experts). Train separate subnets for different tasks, train a router that selects which subnets to activate at inference time. Mixtral is a current open model which I believe implements this.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token).

Re: What happens if we remove 50 percent of Llama?

#40

Post by qeternity »

zwaps wrote: post #17
devsda wrote: post #6

I don't understand LLMs enough to know if this is a silly question or not. Is it possible to build domain specific smaller models and merge/combine them at query/run time to give better response or performance instead of one large all knowing model that learns everything ?

This is called speculative decoding

No, speculative decoding is when you use a smaller draft model to propose tokens and then use the larger target model to verify the proposals. It has got nothing to do with domain specialization.
Post reply on HN