Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Compressing Neural Networks

arxiv.org

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Self-Compressing Neural Networks

#31

Post by eru »

throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

> I think this might be the first step to making neural networks that actually mimic biological brains.

I don't think that's actually a good goal. I suspect the whole term 'neural network' is just misleading and leads to these kinds of misconceptions.

'Neural networks' are mostly just matrix multiplications interleaved with some simple non-linear functions like \x -> max(0, x). Nothing biological about that.

Re: Self-Compressing Neural Networks

#32

Post by eru »

p1esk wrote: post #25
sva_ wrote: post #23

Earlier quoted context omitted.

It would be foolish not to look for inspiration in a system that had billions of years of evolution invested in it.

We already found the inspiration. That’s how we invented neural networks. Now we need to focus on what works.

It's good for some people to keep looking for more inspiration. Not just from brains, but also from all other aspects of the world.

Eg dropout was (allegedly) inspired by our doubled up chromosomes and evolutionary selection.

Re: Self-Compressing Neural Networks

#33

Post by smegger001 »

TheDudeMan wrote: post #20
throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Stop trying to mimic brains. Do what works best for transistors.

except copying biology (neural networks) has worked better than other approaches tried so far (reasoning as search, symbolic and semantic nets, expert systems...) so stick with whats working and we have a working reference model to study and we can build optimized hardware to match if it keeps working better than other methods

Re: Self-Compressing Neural Networks

#34

Post by Salgat »

throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

If you remove the speech centers of the brain all at once, you completely and permanently lose the ability to speak or understand speech. Do the same but very slowly, and the brain is able to compensate, even if you completely lose your original speech centers in the process. NN with pruning is the same thing, where you prune iteratively while retraining to regain most of the lost regressions. If you prune too much all at once, you have to restart from scratch.

Re: Self-Compressing Neural Networks

#35

Post by Version467 »

xpe wrote: post #28
Version467 wrote: post #7

This is super cool. It's surprising to me that it took so long for someone to try this. It seems like such an obvious idea (in hindsight). But I guess that's easy to say now that someone came up with it. If this turns out to work well even for much larger models, then we might see loss functions that incorporate ever more specific performance metrics, conceivably even actual execution times on specific hardware.

There was related work that happened before, as mentioned in the paper.

Whoops, missed that. Thanks.

Re: Self-Compressing Neural Networks

#36

Post by HeatrayEnjoyer »

luckystarr wrote: post #9
tazu wrote: post #8

Earlier quoted context omitted.

Why? Do you think lossless compression is intelligence?

Parents thinking was probably: If you can achieve similar results with a fraction of memory/compute usage then capability at the same hardware level will increase even more.

"Hardware overhang" is the term of art.

My meek opinion is this is obvious. Human-level intelligence requires at most 20 watts and substrate no more complicated than can be constructed from simple organic molecules in a dirty environment.

What is possible with 20 kilowatts and wafer fabricators?

Re: Self-Compressing Neural Networks

#38

Post by dkersten »

p1esk wrote: post #25
sva_ wrote: post #23

Earlier quoted context omitted.

It would be foolish not to look for inspiration in a system that had billions of years of evolution invested in it.

We already found the inspiration. That’s how we invented neural networks. Now we need to focus on what works.

How do we know that current artificial neural networks aren’t the local maximum of modelling, and there isn’t a better model (biologically inspired or otherwise) that we haven’t explored yet?

We need both to work on improving what we have that works, and to explore other avenues and inspirations (both to try entirely new things, and to improve the things we already have working in new ways). I don’t think it wise to throw out what we have working to try again with something biologically inspired, but I also don’t think it wise to say ok we’ve learned enough from biology, let’s focus purely on what we have now, when we don’t understand so much about biological brains, intelligence, and consciousness.

Re: Self-Compressing Neural Networks

#39

Post by Jensson »

nyrikki wrote: post #24
dontwearitout wrote: post #19

Earlier quoted context omitted.

Are dendritic sub-compartments necessary to explicitly model, or does this work just imply that biological neurons are complicated and are better modeled as a multi-layered artificial network, rather than a single simple computational unit? Similarly, do you think that spiking networks are important, or just a specific mechanism used in the brain to transmit information, which dense (or sparse) vectors of floats do i…

If the goal was to create an artificial neural network that better approximated the biological human brain, yes the perceptron model is insufficient. If your goal is to produce a useful model on real hardware and it works...no Remember the constraints of ANNs being universal approximaters (in theory) 1) The function you are learning needs to be continuous 2) Your model is over a closed, bounded subset of R^n 3) The a…

> 1) The function you are learning needs to be continuous

Seems like a bad limitation when you try to model reasoning based on facts and logic, there are many things there that are just true or false and no spectrum to it. There is no "kinda true" in those circumstances, you should only get 1 or 0 and never any value between.

Re: Self-Compressing Neural Networks

#40

Post by Jensson »

octocop wrote: post #37

What advantages does this have over applying neural network compression methods after training?

If you do it during training you automatically check that the compressed model still do what you want after compression, and can even correct some small issues by continuing the training on the smaller model.

Edit: In general the more a compression function understands of what your goals are the better, so it is naturally advantageous to make the compression function look like training since then it is fully aware of what to optimize for.

Post reply on HN