Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Compressing Neural Networks

arxiv.org

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Self-Compressing Neural Networks

#41

Post by szcs »

Version467 wrote: post #7

This is super cool. It's surprising to me that it took so long for someone to try this. It seems like such an obvious idea (in hindsight). But I guess that's easy to say now that someone came up with it. If this turns out to work well even for much larger models, then we might see loss functions that incorporate ever more specific performance metrics, conceivably even actual execution times on specific hardware.

Author here. I actually came up with the idea a long time ago, I first experimented with variants of this in Caffe (before Tensorflow was a thing).

Re: Self-Compressing Neural Networks

#42

Post by szcs »

octocop wrote: post #37

What advantages does this have over applying neural network compression methods after training?

Here is a quick graph I just generated from data I had. Horizontal axis is average bit depth, vertical is accuracy. PTQ is Post Training Quantisation, QAT is Quantisation Aware Training. It's a simple ResNet trained on CIFAR-10, I don't have the resources to train anything bigger.

https://i.imgur.com/yiVLljh.png

Re: Self-Compressing Neural Networks

#43

Post by nwoli »

w-m wrote: post #27

Using as little computational resources (memory and/or FLOPS) as possible as an additional optimization criterion when training NNs is an interesting avenue. I think the current state of pre-trained model families is weird. Take Llama 3.1 or Segment Anything 2: you get tiny/small/medium/larger/huge models, where for each tier the model size was predefined, and they are trained somewhat (completely?) independently. Th…

One elegant approach for this I’ve found is this https://github.com/mit-han-lab/gan-compression They basically train an “all in one” network from which you can extract small or large models afterwards (with optional additional finetuning to improve the selected channel size combinations)

Re: Self-Compressing Neural Networks

#47

Post by dpkingma »

There seems to be some relevant prior work that is not referenced by this paper, such as our work on training sparse neural networks:

https://arxiv.org/abs/1712.01312

Abstract: "We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. [...]"

Re: Self-Compressing Neural Networks

#48

Post by bilsbie »

diimdeep wrote: post #3
bilsbie wrote: post #2

dynamic quantization-aware training that puts size (in bytes) of the model in the loss

I know where you saw that https://x.com/realGeorgeHotz/status/1819963680739512550 > This is one of the coolest papers I've seen in a while. "Self-Compressing Neural Networks" is dynamic quantization-aware training that puts size (in bytes) of the model in the loss! > My implementation (in @__tinygrad__): https://github.com/geohot/ai-notebooks/blob/master/mnist_sel...

True. It seemed like a good, concise summary. Maybe I should have credited him.

Re: Self-Compressing Neural Networks

#49

Post by nyrikki »

Jensson wrote: post #39
nyrikki wrote: post #24

Earlier quoted context omitted.

If the goal was to create an artificial neural network that better approximated the biological human brain, yes the perceptron model is insufficient. If your goal is to produce a useful model on real hardware and it works...no Remember the constraints of ANNs being universal approximaters (in theory) 1) The function you are learning needs to be continuous 2) Your model is over a closed, bounded subset of R^n 3) The a…

> 1) The function you are learning needs to be continuous Seems like a bad limitation when you try to model reasoning based on facts and logic, there are many things there that are just true or false and no spectrum to it. There is no "kinda true" in those circumstances, you should only get 1 or 0 and never any value between.

Perceptrons are binary classifiers, that output 0 or 1, based on a threshold.

While not practical to find or use, any feed forward network supervised is effectively a paramedic linear regression.

Think of an Excel line graph, drawing lines between points, with the above the line being 'true', or when the soma fires.

That is how perceptrons work.

Single layer perceptrons cannot represent linearly inseparable functions like XOR or band pass.

A single biological neurons can use the timing of pulses, band pass, change the rate of pulses etc... before it ever reaches the soma.

Not all problem can be reduced to decision problems and not all of them can use constant depth threshold circuits, which hard attention is.

An LLM can be a very reliable threshold or majority gates as an example, but cannot generalize PARITY.

Basically statistical learning inherited the same limits of statistics.

"This statement is 'False'" is a good paradox to use as a lens.

Post reply on HN