I'm curious - what happens if one prunes the halved model again (if that's possible with the same method), would it start losing accuracy?
I was thinking the same. On HF, I see 4bit gguf of this 2:4 model, and I'm like...that works?
Link?
What happens if we remove 50 percent of Llama?
101–110 of 139 posts
Re: What happens if we remove 50 percent of Llama?
#102Earlier quoted context omitted.
Does this mean that the model will be half the size? If a 32B model@4bit normally requires 16 GB VRAM, at half the size, it could be run @8bit with 16 GB VRAM? Isn't that tradeoff a great improvement? I assume the improved bit precision will more than compensate for the loss related to removal?
There is some improvement going from 4-bit to 8-bit quantization, but if you have VRAM to spare for that, you usually see more benefit from running a 2x larger model at 4-bit. So in scenarios where an LM already fits the existing VRAM budget, I would expect larger models instead. The other thing is that VRAM is used not just for the weights, but also for prompt processing, and this last part grows proportionally as y…
For readability, I recommend reserving "b" for bits, "B" for byte, "p" for parameter.
I assume in your post that "30b" meant 30 billion, or in other words, 30Gp (giga-parameter).
Furthermore is 24Gb of VRAM 24 gigabits (power of 10), or 24 gibibits (power of 2)?
Re: What happens if we remove 50 percent of Llama?
#103Earlier quoted context omitted.
I was thinking the same. On HF, I see 4bit gguf of this 2:4 model, and I'm like...that works?
Link?
Re: What happens if we remove 50 percent of Llama?
#104Earlier quoted context omitted.
Even RX 7600 XT has 16GB
I wonder if a 7600 XT is a cut-down 7800 XT then, because both normal and XT variants of the 6700 and 7700 only have 12GB VRAM. Nonetheless, great info. Sounds like it might be the budget inference king!
Completely different chips; the VRAM differences are from how GDDR can be used, with either 1 or 2 chips on a single 32bit bus, the configuration with 2 chips is called clamshell. The 7800 XT and 7600 XT have same VRAM but the 7800 XT has a 256 bit memory bus while the 7600 XT has a 128 bit memory bus. Meanwhile the 7700 XT with 12 GB is on a 192 bit memory bus.
The workstation edition of GPUs usually do the clamshell configuration so they can easily double the VRAM and ramp up the price by a couple thousand
Re: What happens if we remove 50 percent of Llama?
#105Earlier quoted context omitted.
Given the price tag, I don't think I'd call that "consumer" hardware, but rather "professional" hardware. But perhaps that's just me…
Yeah, I also think that the ~5k price is quite hefty. It's difficult for me to imagine that running sizeable LLMs on commodity/consumer hardware will be possible without another breakthrough in the field. The prices of GPUs I wouldn't expect to fall if technology proves its worthiness.
I can imagine a world where "good enough" GPGPUs become embedded in common chipsets the same way "good enough" regular GPUs are embedded now, but we're definitely not there yet. That said, it was only a few years between the VooDoo cards coming to market and Intel integrated graphics showing up.
Re: What happens if we remove 50 percent of Llama?
#106Earlier quoted context omitted.
What i want to mean is difference between 100% fine person vs Functioning Autist. Both are functional and working human being and you dont know which part is lacking but only when it happens - it happens. Make sense?
I think you don't understand what autism even is. Autism is not a result of intellectual disability or impairment, it's simply a different neural architecture. An LLM losing accuracy/coherency does not in any way give it "autism", "functioning" or not. Please don't use "autism" to essentially mean retardation.
Autism is not one thing. For some, intellectual disability (ID) is not separate from their autism .. it shares the same causes.
For others, ID plays no part. even at the subdiagnostic level.
Re: What happens if we remove 50 percent of Llama?
#107All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…
The fact that you can prune a model will not make it smarter, the wall still stands. I think what explains the wall is the fact that we can't scale organic data exponentially, and we have already covered the most useful types.
Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data only linearly, the efficiency won't be the same.
Synthetic data might help us pad up the training sets, but the most promising avenue I think is to use user-LLM chat logs. Those logs contain real world grounding and human in the loop. Millions of humans doing novel tasks. But that only scales linearly with time, as well.
No way around it - we only once had the whole internet for the first time in the training set. After that it's linear time.
Re: What happens if we remove 50 percent of Llama?
#108Earlier quoted context omitted.
Yeah, I also think that the ~5k price is quite hefty. It's difficult for me to imagine that running sizeable LLMs on commodity/consumer hardware will be possible without another breakthrough in the field. The prices of GPUs I wouldn't expect to fall if technology proves its worthiness.
I can imagine a world where "good enough" GPGPUs become embedded in common chipsets the same way "good enough" regular GPUs are embedded now, but we're definitely not there yet. That said, it was only a few years between the VooDoo cards coming to market and Intel integrated graphics showing up.
We already have something similar in terms of HW accelerators for AI workloads in recent CPU designs but that's not enough.
LLM inference workloads are bound by the compute power, sure, but that's not insurmountable IMO. Much bigger challenge is memory. Not even the bandwidth but just a sheer amount of RAM you need to just load the LLM weights.
Specifically, even a single H100 will hardly suffice to host a mid-sized LLM such as llama3.1-70B. And H100 is ~50k.
If that memory amount requirement is there to stay, and with current LLM transformer architecture it is, then what is really left as an only option for affordable consumer HW are only the smallest and least powerful LLMs. I can't imagine having a built-in GPGPU with 80G of on-die memory. IMHO.
Re: What happens if we remove 50 percent of Llama?
#109All of these smaller model paradigm suggests that we need to incorporate pruning into model training. Neat was one of my favorite algorithms of all time. Same thing with BitNet models which keep showing the information you need is not that much for neural networks. And again, it is same with us, we use much less energy than a regular network so there seems to be immense waste of energy training these models. My intiu…
I seem to recall that there a recent theory paper that got a best paper award, but can't find it. If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space. [This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.] So wi…
[deleted]
Re: What happens if we remove 50 percent of Llama?
#110Earlier quoted context omitted.
Given the price tag, I don't think I'd call that "consumer" hardware, but rather "professional" hardware. But perhaps that's just me…
Yeah, I also think that the ~5k price is quite hefty. It's difficult for me to imagine that running sizeable LLMs on commodity/consumer hardware will be possible without another breakthrough in the field. The prices of GPUs I wouldn't expect to fall if technology proves its worthiness.
You're predicting the price of computer chips will not fall? They're just about the most price-fally truly useful thing in history.