Self-Compressing Neural Networks
arxiv.org
Self-Compressing Neural Networks
1–10 of 59 posts
Re: Self-Compressing Neural Networks
#2dynamic quantization-aware training that puts size (in bytes) of the model in the loss
Re: Self-Compressing Neural Networks
#3dynamic quantization-aware training that puts size (in bytes) of the model in the loss
I know where you saw that
https://x.com/realGeorgeHotz/status/1819963680739512550
> This is one of the coolest papers I've seen in a while. "Self-Compressing Neural Networks" is dynamic quantization-aware training that puts size (in bytes) of the model in the loss! > My implementation (in @__tinygrad__):
https://github.com/geohot/ai-notebooks/blob/master/mnist_sel...
Re: Self-Compressing Neural Networks
#4That is pretty cool. I found a follow up work that applies this technique to LLM: https://konczer.github.io/doc/Poster_EEML23_JozsefKonczer.pd...
Re: Self-Compressing Neural Networks
#5This kind of thing, much more than LLMs, makes me worry about AGI takeoff.
Re: Self-Compressing Neural Networks
#6I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn.
This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the first one I’ve seen where the network is self optimizing.
Re: Self-Compressing Neural Networks
#7This is super cool. It's surprising to me that it took so long for someone to try this. It seems like such an obvious idea (in hindsight). But I guess that's easy to say now that someone came up with it.
If this turns out to work well even for much larger models, then we might see loss functions that incorporate ever more specific performance metrics, conceivably even actual execution times on specific hardware.
Re: Self-Compressing Neural Networks
#8This kind of thing, much more than LLMs, makes me worry about AGI takeoff.
Why? Do you think lossless compression is intelligence?
Re: Self-Compressing Neural Networks
#9This kind of thing, much more than LLMs, makes me worry about AGI takeoff.
Why? Do you think lossless compression is intelligence?
Parents thinking was probably: If you can achieve similar results with a fraction of memory/compute usage then capability at the same hardware level will increase even more.
Re: Self-Compressing Neural Networks
#10I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…
i know nothing of which i speak.... but the theme reminded me of the insect brains, with very relatively few Neurons that manage pretty extraordinary feats.
i guess random evolution pruning happens and if there is no detrimental effect, cheerio.