Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Compressing Neural Networks

arxiv.org

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Self-Compressing Neural Networks

#11

Post by andrewflnr »

luckystarr wrote: post #9
tazu wrote: post #8

Earlier quoted context omitted.

Why? Do you think lossless compression is intelligence?

Parents thinking was probably: If you can achieve similar results with a fraction of memory/compute usage then capability at the same hardware level will increase even more.

It's specifically the fact that the network is directing its own optimization. Which yes, could then potentially be used to get more capability from the hardware, but that's true of manually optimized networks as well. Needing less human help is the... interesting part.

Re: Self-Compressing Neural Networks

#13

Post by andrewflnr »

tazu wrote: post #8
andrewflnr wrote: post #5

This kind of thing, much more than LLMs, makes me worry about AGI takeoff.

Why? Do you think lossless compression is intelligence?

No, but since you mentioned it: https://en.wikipedia.org/wiki/Hutter_Prize

Anyway, OP is about lossy compression. I can't fully follow it but they talk about techniques for mitigating loss later in the paper.

Re: Self-Compressing Neural Networks

#14

Post by svantana »

spacemanspiff01 wrote: post #12

So this was published a year and a half ago? Is there a reason it did not catch on?

It's not really that innovative. As the paper notes, there are several similar previous works. Also, it sounds like they have done a bunch of tweaking to reduce the "irreversible forgetting" specifically for this particular dataset and network, which is not very scientific. Further testing is required to see if this method really has legs.

Re: Self-Compressing Neural Networks

#15

Post by idiotsecant »

tazu wrote: post #8
andrewflnr wrote: post #5

This kind of thing, much more than LLMs, makes me worry about AGI takeoff.

Why? Do you think lossless compression is intelligence?

Compressing understanding (not just information) in a way that uses semantic links in information is a big part of intelligence, I'd say.

Re: Self-Compressing Neural Networks

#16

Post by nyrikki »

throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Unfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO.

As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiveness through additional compression. Being able to control loss of precision is exciting though.

It may help in some cases, especially for practical use cases, but their unaddressed mention of potential problems with noisy loss functions needs to be addressed.

Human biological neurons can do XOR in the dendrites without hitting the soma at all is another example.

If you haven't heard about dendritic compartmentalization and plasticity, here is a paper.

https://www.cell.com/neuron/fulltext/S0896-6273(11)00993-7

> In conclusion our results support the view that experience can drive clustered synaptic enhancement onto neuronal dendritic subcompartments, providing fundamental architecture to circuit development and function

Re: Self-Compressing Neural Networks

#17

Post by derefr »

nyrikki wrote: post #16
throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Unfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO. As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiven…

> reduces ANN expressiveness

But does it? It’s been my hypothesis for a while that every grad-trained NN is hauling around a lot of “nascent” nodes — nodes that were on their way to being useful, but haven’t received enough input yet to actually have their outputs be distinguishable from noise / ever influence the output. Sort of the neuroplastic equivalent of an evolutionary pre-adaptation.

If such nodes exist in NNs, they would be important to decreasing training time to learning new concepts given further training; but if there will be no more training, then they could be pruned for literally no change in expressivity (i.e. the optimality of the NN as an autoencoder of the existing training data.)

Re: Self-Compressing Neural Networks

#18

Post by IIAOPSW »

nyrikki wrote: post #16
throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Unfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO. As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiven…

You piqued my curiosity, so I looked for a paper. I found something tangential but fascinating.

"Naud and Sprekeler (2018) suggest that this could be achieved using a synaptic strategy that facilitates summation for simple action potentials arriving on the basal dendrites and depresses faster burst-like events arriving on the distal tuft"

Oh, its frequency multiplexing with a band pass filter. Same trick the analog phone system used to reduce the amount of wire needed in the network. Same problem, same solution. Convergent evolution.

I wonder if there's ways to do phreaking on neurons.

https://www.sciencedirect.com/science/article/pii/S030645222...

Re: Self-Compressing Neural Networks

#19

Post by dontwearitout »

nyrikki wrote: post #16
throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Unfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO. As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiven…

Are dendritic sub-compartments necessary to explicitly model, or does this work just imply that biological neurons are complicated and are better modeled as a multi-layered artificial network, rather than a single simple computational unit?

Similarly, do you think that spiking networks are important, or just a specific mechanism used in the brain to transmit information, which dense (or sparse) vectors of floats do in artificial neural networks?

Re: Self-Compressing Neural Networks

#20

Post by TheDudeMan »

throwup238 wrote: post #6

I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the fi…

Stop trying to mimic brains. Do what works best for transistors.
Post reply on HN