81–90 of 135 posts
Earlier quoted context omitted.
So sin() could be used instead of tanh, if appropriately shifted and scaled I presume?
You'd at least want to keep it at ±1 once it reaches that value instead of oscillating.
But yeah I wasn't thinking too much about large input values, I presumed clamped inputs, which I guess might not be ideal.
Earlier quoted context omitted.
You'd at least want to keep it at ±1 once it reaches that value instead of oscillating.
I was thinking of a half-period, ie +/- pi/2. But yeah I wasn't thinking too much about large input values, I presumed clamped inputs, which I guess might not be ideal.
ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…
Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.
The strongest statement that might fit here is that they can't learn efficiently. Zero gradients just make learning slower (though I do agree that differentiability is something to strive for).
As a bit of an aside, in practice stochastic versions of algorithms assuming differentiability work on wide ranges of functions, and compositions of poorly behaved functions can be quite nicely.
For a couple [0] concrete examples:
(1) Throw the absolute value function into your favorite gradient/Newton's minimization routine. Blindly using differentiable techniques often works if a sub-gradient technique would work.
(2) Consider minimizing the magnitude of the smallest eigenvalue of the Jacobian matrix of your favorite function. Many of the intermediate components (e.g., trying to derive the eigenvalue with respect to matrix entries) are poorly defined, undefined, or have cusps and other nasty features. The composition is (under mild constraints) differentiable with non-zero gradients.
(3) Consider minimizing the absolute value of a step function. By using a wide difference quotient as an approximation of the derivative and feeding that into optimizers you'll still find the minimum near zero (See (1); it works similarly).
If the composite output is only constant on small regions in the input space (which holds if those neurons are modeling anything non-trivial), you can rig together something close enough to backprop to still learn efficiently.
ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…
To add to that, there's evidence that may mean that memories are stored in DNA, shuffled between neurons in RNA capsids, evolutionary borrowed from retroviruses. That would make each neuron a complicated computer, functionally equivalent to one pc in capability and data storage. "The neuronal gene Arc is essential for long-lasting information storage in the mammalian brain, mediates various forms of synaptic plastici…
>That would make each neuron a complicated computer
Is this not an accepted fact? Each neuron is like a tiny processor in a larger distributed like system. This is why it's impossible to build bioaccurate NNs because each neuron has ~10K connections to various other neurons.
Earlier quoted context omitted.
That’s true, the brain uses a VM to run maths or science, only the best scientists succeed at understanding some of the concepts natively. Same for music: A student runs the music sheet in a VM, and progressively JIT makes the movements native, which allows much faster execution, and which allows building on top of the base layer. Maybe we’re doing it all wrong writing programs in assembler. We should give them to a…
> Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, This is what compilers do. Their input is a program in a more abstract language, either bytecode, an intermediate representation, or a source language. The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidabl…
Earlier quoted context omitted.
There is no need to approximate a ReLu or tanh well. Machine learning is statistical. The accuracy of these functions is not that important ReLu is buggy and has an incorrect activation function for deep learning because it's not continuous everywhere. In practice, it rarely matters. It's chosen only because it's fast to implement buggy function than use someting proper. The exact shape of tanh is not important eithe…
> Tanh is implemented in hardware so it's used. Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.
However tanh is a bit more special than just having the right symmetries. Sigmoid is the correct function to turn an additive value into a probability (range 0 to 1). Tanh is a scaled sigmoid which fulfills the same purpose for the -1 to +1 interval.
I sometimes wonder if clamped linear or exponential functions would work better than tanh/sigmoid in places where they're currently used (like LSTM/GRU gates).
ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…
Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…
I think that's largely known, depending of course on how you define "cognition".
Huge tracts (I don't have numbers) of the cortex are dedicated to things like vision, motor control, etc... Those aren't "cognition" as generally understood, and there are many stroke victims out there who can testify (like, actually "testify", in the sense of using their brain to explain it to you) to the fact that they can no longer see, or move their left side, etc... Their "cognition" is not impaired.
It gets fuzzier with things like speech and recognition, which also have dedicated real estate but are, kinda, "para-cognition" tasks.
Really, yes: you can have a "thinking" engine with a tiny fraction of the computation power of the human brain. I think most folks agree with that. The broader question is that with so limited an I/O structure: what is there for it to think about?
> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…
On the other hand, a single cell much simpler than a neuron (any bacteria) is able to perform significantly more complex calculations than any ANN we've tried so far (successfully interacting with an environment to move and find food). Comparing these kinds of disparate tasks for "computational power levels" between vastly different architectures one of which we're not even close to understanding is generally pretty…
> successfully interacting with an environment to move and find food
Yet the strategies they implement are equivalent to suprisingly simple to implement things, such as PI control or gradient descent.
Earlier quoted context omitted.
Their pretrained 128-wide net has a total of 9.2 million parameters: https://www.kaggle.com/selfishgene/single-neuron-as-deep-net...
9.2 million parameters per neuron. There an estimated 86 billion neurons in the human brain [0] and 19 billion in the neocortex [1]. That means that, for this strategy to emulate a human brain or neocortex would require 791 quadrillion parameters or 175 quadrillion parameters, respectively. The largest ANN built so far, GPT-3, has 175 billion parameters [2]. We are 6 orders of magnitude from being able to pull it off…
Some of those weights presumably go to reproducing highly-conserved features, like the kinetics of particular ion channels. These are "tied" via the genome, in the sense that there's one KCNC1 gene, but millions of neurons express the ion channel it encodes.
On the other hand, this model is also missing all sorts of other interactions: hormones and other neuromodulators, ephaptic coupling, etc.
It's so complicated I would venture that no one even has a reasonable guesstimate of how close we are, beyond "Not very."
> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…
There is no need to approximate a ReLu or tanh well. Machine learning is statistical. The accuracy of these functions is not that important ReLu is buggy and has an incorrect activation function for deep learning because it's not continuous everywhere. In practice, it rarely matters. It's chosen only because it's fast to implement buggy function than use someting proper. The exact shape of tanh is not important eithe…
Similarily there might not a need to emulate neurons well to get the circuits in the brain to work. However when someone makes arguments that neurons are equivalent x artifical neurons it is necessary to choose a bound for comparison (fe. L2 error of activation) for the emulations you compare.