Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

arxiv.org

Post reply on HN
Print view View original

31–40 of 66 posts

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#31

Post by roomey »

napo wrote: post #14
dougmwne wrote: post #9

Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.

It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.

It must give an evolutionary advantage, or we wouldn't sleep.

It may be hard to pin point exactly what advantage, but as we do it, it must have given us an advantage!

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#33

Post by dougmwne »

roomey wrote: post #31
napo wrote: post #14

Earlier quoted context omitted.

It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.

It must give an evolutionary advantage, or we wouldn't sleep. It may be hard to pin point exactly what advantage, but as we do it, it must have given us an advantage!

Especially considering that it is so widespread in nearly every creature with a brain. And it’s not simply a period of motionless energy conservation but has very specific neural patterns. The science is definitely zeroing in on a connection to learning.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#34

Post by gfodor »

icyfox wrote: post #20

Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are: - Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.…

Reduction in working memory for sparse models seems pretty huge.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#35

Post by NM_Ricky »

For anyone interested in SparseGPT, on May 25th, the author of the SparseGPT paper will show you how you can download an optimized and open-sourced LLM and run it on CPUs at GPU speeds using DeepSparse.

Confirm your spot: https://neuralmagic.com/unlock-faster-and-more-efficient-lan...

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#36

Post by MacsHeadroom »

vessenes wrote: post #17

This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered…

SparseGPT-for-LLaMA[0] exists. Pruning more than 30% of weights of 30B starts to show significant perplexity losses. 50% is a total disaster, while it was not for OPT or BLOOM. So your intuition seems to be good here.

[0] https://github.com/AlpinDale/sparsegpt-for-LLaMA

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#37

Post by MacsHeadroom »

babyshake wrote: post #4

It looks like by pruning by a factor of 0.5, you reduce the size of the model by 50%? In practice, what is the expected observed change in the output before and after pruning?

>what is the expected observed change in the output before and after pruning?

The expected and observed change is virtually none. That's the whole point!

Notably, quantizing weights from 16bit weights to 4bit weights (reducing the size by 75%) also has almost no change in output quality when using modern algorithms like GPTQ.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#39

Post by CuriouslyC »

icyfox wrote: post #21
icyfox wrote: post #20

Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are: - Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.…

Another random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of the layer. In theory it's possible to feed actual datasets into these models as well, which could be another way to get a sparse model that's more acutely optimized towards one task. The original model produces the X activations on each layer, and these are u…

Statisticians have been using L1 regularization to estimate sparse models for a while, it seems reasonable to assume that you could fine tune the model on a data set while also pushing weak parameters to zero in a natural way in this domain as well.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#40

Post by generalizations »

MacsHeadroom wrote: post #36
vessenes wrote: post #17

This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered…

SparseGPT-for-LLaMA[0] exists. Pruning more than 30% of weights of 30B starts to show significant perplexity losses. 50% is a total disaster, while it was not for OPT or BLOOM. So your intuition seems to be good here. [0] https://github.com/AlpinDale/sparsegpt-for-LLaMA

Has anyone figured out what the optimal pruning is for 65b? I don't really know what that matrix in your link is saying, but it didn't seem to show optimal pruning.
Post reply on HN