This is amazing insight, thanks for sharing!
This is your first comment in six years, what is amazing about it?
The Smallest Brain You Can Build: A Perceptron in Python
51–60 of 86 posts
Re: The Smallest Brain You Can Build: A Perceptron in Python
#52In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE That was the first thing that came to mind when I read "the smallest brain you can build ". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
Re: The Smallest Brain You Can Build: A Perceptron in Python
#53If you want to learn the fundamentals of ML I recommend a book, such as Deep Learning: Foundations and Concepts by Chris Bishop. If you insist on staying online, one option is https://course.fast.ai/ If you don't know ML I don't think you're going to learn much through ad hoc demos.
I remember sitting in the senior study lounge reading the previous Bishop book and implementing the perceptron from it, 22 years ago: https://github.com/llimllib/personal_code/blob/945b017b2915c...
(before numarray and numpy merged!)
Re: The Smallest Brain You Can Build: A Perceptron in Python
#54In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE That was the first thing that came to mind when I read "the smallest brain you can build ". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
This doc on Ilya Sutskever & Geoffrey Hinton gives a great background on the progression of deep learning over the past decades [0].
Tl;dr - compute was the bottleneck.
I am not associated with this channel/video, just love it. I’ve shared it here before.
Re: The Smallest Brain You Can Build: A Perceptron in Python
#55In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE That was the first thing that came to mind when I read "the smallest brain you can build ". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
> why there wasn't more progress on neural nets being used for many things prior to the 21st century
They were simply too computationally expensive to train for the limited things they could do. It wasn’t until we had the ability to train large neural networks on commodity hardware that things really took off.
Re: The Smallest Brain You Can Build: A Perceptron in Python
#56The IF statement is the root creator of software programming. It has the ability to compare two values against each other and branch out to blocks of instructions. So it is perceiving (reading), decision making and routing - all that which differentiate life from inanimate objects. The AI agents perform the exact same loop, by delegating the first two steps to a model. Going further backwards, the transistor (or a PN…
> Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) (..)
Back up a bit please! Analog computing is a thing. And it isn't even new - not by a long shot.
There are good reasons why practically all computing today is the digital kind. But electronic 'equivalents' of neural nets is one area where analog might make sense. Adding inputs can be as simple as a bunch of resistors + a transistor. Even on modern silicon nodes, that might be a more efficient setup than digital inputs, N-bit adders/multipliers etc. Not saying that's the case, and AI hardware should be based on analog circuitry. But it could be, and perhaps found to be practical.
Re: The Smallest Brain You Can Build: A Perceptron in Python
#57Earlier quoted context omitted.
The alternative IF expression or statement, in the form introduced by John McCarthy in 1958 (which he used in early 1959 to define his version of AND, OR and NOT), is one of the possible primitives for computation. There exist several equivalent sets of primitive operations. While the sets containing only NAND or only NOR, or both AND and NOT or both OR and NOT are more notorious, these logical operations are more ab…
This reminds me of a book, whose title I forget, about creating a compact set of knowledge that will enable us to "rebuild civilization from scratch" in case of a future/fictional post-collapse scenario. I sometimes wonder, given the immense complexity and global supply-chain dependencies of our computing stack, whether we could bootstrap from first principles something of equivalent power and expressivity, but order…
I think if civilization collapses they'll have other priorities, and very little benefit to get from the effort required for such simpler computing, for a good while...
Re: The Smallest Brain You Can Build: A Perceptron in Python
#58Okay, it’s conscious. But can it run doom? I rest my case.
I know you're joking, but if you really wanted to, you can if you have a network of these, since you can build NAND gates from perceptrons. If you have NAND gates, then you can build any other gates from these and then you can build a computer [1].
Re: The Smallest Brain You Can Build: A Perceptron in Python
#59I think it should be quite obvious that perceptrons are far from the smallest units that are capable of learning. They store many bytes of information, require a non-local update process, need numeric (i.e. symbolic) inputs and involve relatively complex computations. You can go much simpler. For example:
https://medium.com/@VictorBanev/the-simplest-learning-machin...
This is a description of a 5-line algorithm that learns and stores approximate probability of an event using just 1 byte of persistent memory.
Re: The Smallest Brain You Can Build: A Perceptron in Python
#60In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE That was the first thing that came to mind when I read "the smallest brain you can build ". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
The first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile."
Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both):
https://github.com/dbrll/Xortran (backprop on IBM 1130, 60s)
https://github.com/dbrll/ATTN-11 (Transformer on PDP-11, 70s)
What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.