Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a neural network's clever solution to binary addition

cprimozic.net

Post reply on HN
Print view View original

11–20 of 160 posts

Re: Reverse engineering a neural network's clever solution to binary addition

#11

Post by agapron1 »

greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results.

Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

Re: Reverse engineering a neural network's clever solution to binary addition

#12

Post by nerdponx »

I liked this article a lot, but

> One thought that occurred to me after this investigation was the premise that the immense bleeding-edge models of today with billions of parameters might be able to be built using orders of magnitude fewer network resources by using more efficient or custom-designed architectures.

Transformer units themselves are already specialized things. Wikipedia says that GPT-3 is a standard transformer network, so I'm sure there is additional room for specialization. But that's not a new idea either, and it's often the case that a after a model is released, smaller versions tend to follow.

Re: Reverse engineering a neural network's clever solution to binary addition

#14

Post by Jensson »

agapron1 wrote: post #11
greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

Yeah, the current solution is similar to overfitting, this wont generalize to harder math where the operation doesn't correspond to the activation function of the network.

Re: Reverse engineering a neural network's clever solution to binary addition

#15

Post by perihelions »

mananaysiempre wrote: post #9
juunge wrote: post #5

Awesome reverse engineering project. Next up: reverse engineer a NN's solution to Fourier Transform!

The Fourier transform is also linear, so the same solution should work. No clue if an NN would find it though.

What's an analog implementation of a Fourier transform look like? It sounds interesting!

Re: Reverse engineering a neural network's clever solution to binary addition

#16

Post by torginus »

agapron1 wrote: post #11
greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.

Re: Reverse engineering a neural network's clever solution to binary addition

#17

Post by bradrn »

perihelions wrote: post #15
mananaysiempre wrote: post #9

Earlier quoted context omitted.

The Fourier transform is also linear, so the same solution should work. No clue if an NN would find it though.

What's an analog implementation of a Fourier transform look like? It sounds interesting!

It looks like Albert Michelson's Harmonic Analyzer: see https://engineerguy.com/fourier/ (and don’t miss the accompanying video series! It’s pretty cool.)

Re: Reverse engineering a neural network's clever solution to binary addition

#19

Post by GTP »

torginus wrote: post #16
agapron1 wrote: post #11

Earlier quoted context omitted.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.

Yes, but the adder that the NN came up with has no carry.

Re: Reverse engineering a neural network's clever solution to binary addition

#20

Post by sebzim4500 »

agapon wrote: post #4

Very interesting. But I missed how the network handled the overflow. Spotted one small typo: digital to audio converter should be digital to analog.

It converted the inputs to analog so there isn't really a notion of 'overflow'.
Post reply on HN