This suggests that the effective number of parameters is far lower than the nominal number. My head canon for neural networks as overparametrized models still holds.
SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
51–60 of 66 posts
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#52The robustness with which these models can be quantized, and now trimmed makes one think if they could be easily implemented some form of analog (or optical) hardware.
Isn't the inspiration behind these models our own analog hardware?
not sure i would call neurons analog, they are very nonlinear and capricious beasts.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#53Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#54Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#55This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered…
Could you explain what you mean by "Chinchilla under-trained" or "Chinchilla over-trained"? I assume it refers to some measure of trained-ness, but Googling yielded nothing relevant.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#56This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered…
Could you explain what you mean by "Chinchilla under-trained" or "Chinchilla over-trained"? I assume it refers to some measure of trained-ness, but Googling yielded nothing relevant.
My memory says that there’s a “Chinchilla” paper showing how to make the best model with a given training budget. There’s a trade-off between the amount of training data and the size of the model itself. Chinchilla under-training would mean that the model is too big for the amount of training data used. Llama is Chinchilla over-trained in that there is a ton of data relative to the small size of the model.
Note that this is still desirable for inference because you want the most possible training on whatever model you can actually fit in your memory.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#57Earlier quoted context omitted.
Isn't the inspiration behind these models our own analog hardware?
not sure i would call neurons analog, they are very nonlinear and capricious beasts.
analog and linear are not the same thing.
> adjective: analog
> relating to or using signals or information represented by a continuously variable physical quantity such as spatial position, voltage, etc.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#58Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.
It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.
You've, intentionally or not, ignored a whole body of research. An extremely cursory dive into sleep will show all sorts of functional reasons, related to memory formation.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#59It looks like by pruning by a factor of 0.5, you reduce the size of the model by 50%? In practice, what is the expected observed change in the output before and after pruning?
>what is the expected observed change in the output before and after pruning? The expected and observed change is virtually none. That's the whole point! Notably, quantizing weights from 16bit weights to 4bit weights (reducing the size by 75%) also has almost no change in output quality when using modern algorithms like GPTQ.
What are the drawbacks of quantizing though?
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#60This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered…
Could you explain what you mean by "Chinchilla under-trained" or "Chinchilla over-trained"? I assume it refers to some measure of trained-ness, but Googling yielded nothing relevant.
It's from this paper - https://arxiv.org/abs/2203.15556
Like the sibling comment said - the proportion of training tokens to parameter size is very important, and there's a certain threshold needed to be met for it to be "fully trained".
Usually you have a fixed amount of compute (budget/time essentially) - and in that case you want to pick the largest parameter count that you can fully train, and not the largest parameter count your hardware can support and then train that for less time.
tl;dr - Small models with training over the chinchilla threshold can out perform large models that are undertrained
EDIT: Figure 2 page 5, and Table 3 page 8 - might be worth checking out.