The robustness with which these models can be quantized, and now trimmed makes one think if they could be easily implemented some form of analog (or optical) hardware.
Isn't the inspiration behind these models our own analog hardware?
SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
11–20 of 66 posts
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#12The robustness with which these models can be quantized, and now trimmed makes one think if they could be easily implemented some form of analog (or optical) hardware.
Perhaps even by biological cells!
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#13Earlier quoted context omitted.
How would this be any different from running one of the lower parameter models?
It says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.
I wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#14Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.
It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI.
It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#15To copy a reddit meme: text-generation-webui plugin when?
But seriously, this seems like an incredible upgrade.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#16Earlier quoted context omitted.
It says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.
I wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
Good open questions. I suspect we'll see models distilled and compressed down to retain most of their "common sense", "core knowledge", and reasoning ability, while stripping out the gigabytes of random trivia most people will never need.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#17This is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive.
Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered by a sparsification like this? Or do you need a special implementation to not malloc out all the memory implied by each tensor layer?
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#18To copy a reddit meme: text-generation-webui plugin when? But seriously, this seems like an incredible upgrade.
Better yet, text-generation-webui-docker when?
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#19If the abstract is accurate, then I'm very, very excited to try this on LLaMA 65B. We are tantalizingly close to ChatGPT performance parity on consumer hardware.
Hopefully this lowers the cost of doing instruct fine tuning on the larger models, and we see a Vicuna like model based on LLaMA 65B soon. This is exciting folks.
Re: SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
#20Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are:
- Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.3B model
- The core issue to scale is time to calculate the Hessian during prune analysis (effectively a matrix of second-order derivatives, famously computationally intense to calculate)
- They follow the existing literature and use a local approach to each layer. By doing this (and doing it well), it can preserve the input/output contract for surrounding layers, which makes the whole thing paralellizable across machines
- Their solution approximates reconstruction loss by approximating a quadratic loss and then running a OBS update (with a few other optimizations on ordering and iteration on the side)
I'm particularly excited for these smaller models, mostly for inference efficiency gains in realtime applications. The general con of weight pruning is they still require incredibly large training clusters / investment in training resources upfront to get the original parameter weight. But if the lottery ticket hypothesis holds true, this might be the best way we have at the moment to get models with same performance and lower longterm operational costs.