81–90 of 160 posts
https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec...
One interesting thing is that this network similarly does addition using a Fourier transform.
These stories remind me of a story from Discover Magazine https://www.discovermagazine.com/technology/evolving-a-consc... A researcher was using a process to "evolve" a FPGA and the result was a circuit that was super efficient but worked in ways that were unexpected: part of the circuit seemed unconnected to the rest but if removed the whole thing stopped working and it would only work at a specific temperature.
Impressive. Am I right that what is really going on here is that the network is implementing the "+" operator by actually having the CPU of the host carrying out the "+" when executing the network? I.e., the network converts the binary input and output to floating point, and then it is the CPU of the host the network is running on that really does the addition in floating point. So usually one does a bunch of FLOPs t…
My intuition is as follows: if I were to train this network with pencil, paper and a slide rule, I'd expect the same result. Addition (or maybe rather integration) is embedded in the abstract structure of a neural network as a computation artifact.
Sure, specifics of the substrate may "leak through" - e.g. in the pen-and-paper case, were I to round everything to first decimal space, or in the computer case, was the network implemented with 4-bit floats, I'd expect it not to converge because of loss of precision range (or maybe figure out the logic gate solution). But if the substrate can execute the mathematical model of a neural network to sufficient precision, I'd expect the same result to occur regardless of whether the network is run on paper, on a CPU, an fluid-based analog computer, or a beam of light and a clever arrangement of semi-transparent plastic plates.
Earlier quoted context omitted.
Very easy to design a radio component into electronics, much harder to design it out.
I’ve had all kinds of cheap electronics that came with unadvertised radio features for free.
The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a hug…
Not too surprising. Each layer can multiply each input by a set of weights, and add them up. That's all you need, in order to convert from base 2 to base 10. In base 2, we calculate the value of a set of digits by multiplying each by 2^0, 2^1, ..., 2^n. Each weight is twice the previous. The activation function is just making it easier to center those factors on zero to reduce the magnitude of the regularization term…
Earlier quoted context omitted.
One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a hug…
Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or pla…
Earlier quoted context omitted.
That's an accurate enough description of what's going on here, yeah, but I'm very curious if this could be implemented in hardware. What we've got is an inexact not-quite-binary adder, but one that's potentially smaller and faster than the regular binary ones.
Why it works here is that the "analog" representation is not influenced by noise, because it's simulated on digital hardware. On the other hand, why we use digital hardware precisely because it's robust against the noise present in the hardware analog circuits.
I wonder, is there ever a case analog-on-digital is better to work with as an abstraction layer, or is it always easier to work with digital signals directly?