Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

arxiv.org

Post reply on HN
Print view View original

21–30 of 66 posts

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#21

Post by icyfox »

icyfox wrote: post #20

Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are: - Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.…

Another random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of the layer. In theory it's possible to feed actual datasets into these models as well, which could be another way to get a sparse model that's more acutely optimized towards one task. The original model produces the X activations on each layer, and these are used as the optimization criteria for the pruned version.

It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#22

Post by bitL »

Der_Einzige wrote: post #10
gorkish wrote: post #7

Very good result, and awesome to see such great progress happen so fast. Quantizing/pruning/deduplicating/compressing models and embeddings is still a vast orchard of low hanging fruit. I personally think there are still quite a few multiple-orders-of-magnitude scale opportunities to accelerate inference, and we are fortunate to have strong economic incentives aligned with the problem.

So much low hanging fruit for those willing to pick it. It's a great time to be an LLM researcher.

Yeah, writing a thesis on it right now; this + adapters give so many options to play with and Meta was nice to give me access to their research models.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#23

Post by teej »

icyfox wrote: post #20

Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are: - Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.…

This is a great breakdown. I don’t know much about LLM internals but I could follow this easily.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#24

Post by avereveard »

bick_nyers wrote: post #5
avereveard wrote: post #3

wow this could make the fabled 65 billion parameter llama sparsed and pruned runnable on a 3060

How would this be any different from running one of the lower parameter models?

Larger models seem to handle much better introspection, makes for better backend for sourced knowledge extraction

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#25

Post by amelius »

sva_ wrote: post #6
bick_nyers wrote: post #5

Earlier quoted context omitted.

How would this be any different from running one of the lower parameter models?

It says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.

This makes me wonder what would happen if you took the sparse model, reset its weights, then trained it with the original training data.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#26

Post by valine »

dougmwne wrote: post #9

Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.

Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#27

Post by sva_ »

amelius wrote: post #25
sva_ wrote: post #6

Earlier quoted context omitted.

It says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.

This makes me wonder what would happen if you took the sparse model, reset its weights, then trained it with the original training data.

Very interesting idea. I'd hypothesize that it won't achieve the same(ish) accuracy, and that pruning might be required (similar to how humans go through a heavy pruning phase at an early age[0]). Would be worth setting up an experiment on a smaller scale.

As some other commentator stated, there's currently a lot of low hanging fruit in optimizing NN.

0. https://en.m.wikipedia.org/wiki/Synaptic_pruning

Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

#29

Post by dougmwne »

valine wrote: post #26
dougmwne wrote: post #9

Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.

Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.

Extremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product scale. A few extra million for training is expensive, but then consider what it costs to run inference for something like Bing for 100 million daily active users.

If they can develop new methods to “overtrain” these models they will get more bang out of the smaller parameter model buck.

Post reply on HN