Author here, I just noticed this. If you have any questions I can try answering them.
Do you have plans to apply this on a small open LLM as a POC to show that there is no loss of performance?
Yes, LLMs are planned too.
Self-Compressing Neural Networks
51–59 of 59 posts
Re: Self-Compressing Neural Networks
#52(Jan 2023)
Re: Self-Compressing Neural Networks
#53Earlier quoted context omitted.
We already found the inspiration. That’s how we invented neural networks. Now we need to focus on what works.
How do we know that current artificial neural networks aren’t the local maximum of modelling, and there isn’t a better model (biologically inspired or otherwise) that we haven’t explored yet? We need both to work on improving what we have that works, and to explore other avenues and inspirations (both to try entirely new things, and to improve the things we already have working in new ways). I don’t think it wise to…
let’s focus purely on what we have now
That’s not what I said.
Try different things, choose what works, as opposed to trying to imitate biology for the sake of imitating biology.
Re: Self-Compressing Neural Networks
#54Earlier quoted context omitted.
Stop trying to mimic brains. Do what works best for transistors.
It would be foolish not to look for inspiration in a system that had billions of years of evolution invested in it.
It's also full of legacy cruft. We can barely see three colors because some of our ancestors were nocturnal and we never regained the lost receptors [0] [1]. Look at that beautiful bird coverage of the spectrum, look at that awful barely orthogonal human M and L receptor. The blood vessels obstruct the retina in vertebrates! Methanogenesis hasn't been invented outside of Archaea! Also brain with higher cognitive functions (which everyone demands from these neural networks) is barely a couple million years old. Evolution works with copy-paste, rotate and change-single-character, and every single edit has to be compilable and viable.
[0] https://en.wikipedia.org/wiki/File:BirdVisualPigmentAbsorban... [1] https://en.wikipedia.org/wiki/File:Cones_SMJ2_E.svg
Re: Self-Compressing Neural Networks
#55Using as little computational resources (memory and/or FLOPS) as possible as an additional optimization criterion when training NNs is an interesting avenue. I think the current state of pre-trained model families is weird. Take Llama 3.1 or Segment Anything 2: you get tiny/small/medium/larger/huge models, where for each tier the model size was predefined, and they are trained somewhat (completely?) independently. Th…
One elegant approach for this I’ve found is this https://github.com/mit-han-lab/gan-compression They basically train an “all in one” network from which you can extract small or large models afterwards (with optional additional finetuning to improve the selected channel size combinations)
Ahh that's an interesting paper I must of missed that one - thanks for the link. I think another paper that recently got a lot of hype has been the Matroyshka representation learning paper -- essentially training models with different parameters and output embedding sizes at the same time, basically distillation during training rather than post-training (https://arxiv.org/abs/2205.13147).
Re: Self-Compressing Neural Networks
#56Author here, I just noticed this. If you have any questions I can try answering them.
I don't think I quite understand how to apply your technique to gpus, which have discrete formats like fp4, fp8, fp16, int8, etc. Can you shed some light on that?
Re: Self-Compressing Neural Networks
#57Beautiful idea.
My take on it: I find it difficult to generalize the notion of layer removal when the bit depth of that layer goes to zero. It's wouldn't be straight forward although the authors provide equation 5. It feels like lot of information is missing in this work to even reproduce it. And authors do only 1 case study.
I believe some implementation is required to understand the authors completely. Example, optimizer modification for layer when it is removed in training.
Re: Self-Compressing Neural Networks
#58Blog post on the subject by author https://blog.imaginationtech.com/self-compressing-neural-net...
Re: Self-Compressing Neural Networks
#59Earlier quoted context omitted.
How do we know that current artificial neural networks aren’t the local maximum of modelling, and there isn’t a better model (biologically inspired or otherwise) that we haven’t explored yet? We need both to work on improving what we have that works, and to explore other avenues and inspirations (both to try entirely new things, and to improve the things we already have working in new ways). I don’t think it wise to…
let’s focus purely on what we have now That’s not what I said. Try different things, choose what works, as opposed to trying to imitate biology for the sake of imitating biology.
I think we should imitate biology for the sake of imitating biology, though. Alongside other approaches.
Right now, we know that biology works, because of animal and human intelligence. We don’t yet know if our other approaches have the ability to eventually lead to that.