Is this going to change how 8-but adders are implemented in chips?
No, there was a time where we had analog calculators but there are good reasons why digital electronics won in the computing space.
Reverse engineering a neural network's clever solution to binary addition
21–30 of 160 posts
Re: Reverse engineering a neural network's clever solution to binary addition
#22Earlier quoted context omitted.
An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.
A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.
It's not harder computationally but finding that solution with SGD might be hard. I'd be really interested in the results.
Re: Reverse engineering a neural network's clever solution to binary addition
#23The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Yes I just read it too. It appears to have been discussed here multiple times:
https://hn.algolia.com/?q=bitter+lesson
I'd say it has aged well and there are probably lots of new takes on it based on some of the achievements of the last 6 months.
At the same time, I don't agree with it. To simplify, the opposite of "don't over-optimize" isn't "it's too complicated to understand so never try". It is thought provoking though
Re: Reverse engineering a neural network's clever solution to binary addition
#24I liked this article a lot, but > One thought that occurred to me after this investigation was the premise that the immense bleeding-edge models of today with billions of parameters might be able to be built using orders of magnitude fewer network resources by using more efficient or custom-designed architectures. Transformer units themselves are already specialized things. Wikipedia says that GPT-3 is a standard tra…
Transformers are specialized to run on our hardware, whereas the article is suggesting architectures which are specialized for a specific task.
Re: Reverse engineering a neural network's clever solution to binary addition
#25Interesting idea to reverse engineer the network. Are there other sources that have done this?
Maybe not exactly what you had in mind, but is a lot of literature in general on trying to extract interpretations from neural network models, and to a lesser extent from other complicated nonlinear models like gradient boosted trees. Somewhat famously, you can plot the weight activations on a heatmap from a CNN for image processing and obtain a visual representation of the "filter" that the model has learned, which…
Yes, more generally this entails looking at the numbers learned by a model not as singular "weights" (which doesn't scale as the model gets larger) but more as a way of approximating a non-parametric representation (i.e. something function-like or perhaps even more general). There's a well-established field of variously "explainable" semi-parametric and non-parametric modeling and statistics, which aims to seamlessly scale to large volumes of data and modeling complexity much like NN's do.
Re: Reverse engineering a neural network's clever solution to binary addition
#26I liked this article a lot, but > One thought that occurred to me after this investigation was the premise that the immense bleeding-edge models of today with billions of parameters might be able to be built using orders of magnitude fewer network resources by using more efficient or custom-designed architectures. Transformer units themselves are already specialized things. Wikipedia says that GPT-3 is a standard tra…
[deleted]
Re: Reverse engineering a neural network's clever solution to binary addition
#27The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
I think there is no doubt that there must be more efficient model architectures out there, take for example the sample efficiency of GPT-3:
> If you think about what a human, a human probably in a human’s lifetime, 70 years, processes probably about a half a billion words, maybe a billion, let’s say a billion. So when you think about it, GPT-3 has been trained on 57 billion times the number of words that a human in his or her lifetime will ever perceive.[0]
0. https://hai.stanford.edu/news/gpt-3-intelligent-directors-co...
Re: Reverse engineering a neural network's clever solution to binary addition
#28Reminded me of the 0xFFFF0000 0xFF00FF00 0xF0F0F0F0 0xCCCCCCCC 0XAAAAAAAA trick, the use of which I don't currently recall...
edit: several, but probably not very relevant https://graphics.stanford.edu/~seander/bithacks.html
Re: Reverse engineering a neural network's clever solution to binary addition
#29Earlier quoted context omitted.
The Fourier transform is also linear, so the same solution should work. No clue if an NN would find it though.
What's an analog implementation of a Fourier transform look like? It sounds interesting!
Depends on what kind of “analog” you want.
In an NN context, given that you already have “transform with a matrix” as a primitive, probably something very much like sticking a https://en.wikipedia.org/wiki/DFT_matrix somewhere. (You are already extremely familliar with the 2-input DFT, for example: it’s the (x, y) ↦ (x+y, x−y) map.)
If you want a physical implementation of a Fourier transform, it gets a little more fun. A sibling comment already mentioned one possibility. Another is that far-field (i.e. long-distance; “Fraunhofer”) diffraction of coherent light on a semi-transparent planar screen gives you the Fourier transform of the transmissivity (i.e. transparency) of that screen[1]. That’s extremely neat and covers all textbook examples of diffraction (e.g. a finite-width slit gives a sinc for the usual reasons), but probably impractical to mention in an introductory course because the derivation is to get a gnarly general formula then apply the far-field approximation to it.
A related application is any time the “reciprocal lattice” is mentioned in solid-state physics; e.g. in X-ray crystallography, what you see on the CRT screen in the simplest case once the X-rays have passed through the sample is the (continuous) Fourier transform of (a bunch of Dirac deltas stuck at each center of) its crystal lattice[2], and that’s because it’s basically the same thing as the Fraunhofer diffraction in the previous paragraph.
Of course, the mammalian inner ear is also a spectral analyzer[3].
[1] https://en.wikipedia.org/wiki/Fourier_optics#The_far_field_a...
[2] https://en.wikipedia.org/wiki/Laue_equations
[3] https://en.wikipedia.org/wiki/Basilar_membrane#Frequency_dis...
Re: Reverse engineering a neural network's clever solution to binary addition
#30The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
I think there is no doubt that there must be more efficient model architectures out there, take for example the sample efficiency of GPT-3: > If you think about what a human, a human probably in a human’s lifetime, 70 years, processes probably about a half a billion words, maybe a billion, let’s say a billion. So when you think about it, GPT-3 has been trained on 57 billion times the number of words that a human in h…
Humans are not trianed on words.