81–90 of 296 posts
"Here's what you should remember: the only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data." This statement has a few problems - there is no real reason to interpret the transforms as geometric (they are fundamentally just processing a bunch of numbers into other numbers, in what sense is this geome…
My qualm with this article is disappointingly poorly backed up. The author makes claims, but does not justify those claims well enough to convince anyone but people who already agree with him. In that sense, this piece is an opinion piece, masquerading as a science. > This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another. All it c…
Earlier quoted context omitted.
To be fair, we do understand how neurons work, at least on a singular level. Perceptrons model that quite well.
Implementing a basic perceptron classifier is an undergrad homework assignment. Biological modeling of neurons is a work of decades: http://www.genesis-sim.org/ https://www.neuron.yale.edu/neuron/what_is_neuron
The fact we now give this to undergrads as homework suggests that there was some value to this idea.
If the deep learning network has enough layers, then can't it start incorporating "abstract" ideas common to any learning task? E.g. could we re-use some layers for image/speech recognition & NLP?
But yeah, I could see feeding the output of an array of sub-networks into a parent network. So think one NN for vision, one for hearing, etc, etc, all of those outputs feed into a parent level network that could be your abstraction network that deals with making executive level decisions.
The Computational Cognitive Neuroscience Lab has been studying this topic for decades and has an online textbook here:
http://grey.colorado.edu/CompCogNeuro
The "emergent" deep learning simulator is focused on using these kinds of models to model the brain:
As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…
I also believe that researches have responsibility to outline the limits of their own algorithms in research papers. (For example, presenting examples that aren't recognized or data sets on which the approach doesn't work at all.) That is valuable information and they almost certainly have it at the time of publication.
Earlier quoted context omitted.
This is exactly my point - the danger of "anthropomorphization" lies in taking the brain analogy too far. That is, there shouldn't necessarily be a link between research in neuroscience and advances that make deep learning models more accurate. The tasks are completely different (human learning vs. minimizing a loss function), and it's important for researchers in both fields - neuroscience and AI - to keep that in m…
However, there definitely are analogies! E.g. early work in convnets was inspired by the architecture of cat brains. I think the fields have useful things to say to each other, but we're getting over a (maybe justified) taboo in talking about machine learning methods being biologically inspired.
1) Hubel and Wiesel discover simple and complex cells in cat's V1 in the 60's. They came up with an ad hoc explanation that somehow the complex cells "pool" among many simple cells of the same orientation. No one to date knows how such pooling would be accomplished (that selects exactly simple cells of similar orientation and different phase, not vice versa), or whether that pooling is only on V1 or elsewhere in the cortex.
2) Fukushima expanded that ad hoc model into neocognitron in 80's, though there is exactly zero evidence for similar "pooling" in higher cortical areas. In fact, higher cortical areas are essentially impossible to disentangle and characterize even today.
3) Yann Lecun took neocognitron and made a convnet which worked OK for MNIST in the late 80's. Afterward the thing was forgotten for many years.
4) Some few years ago Hinton and some dude who could write good GPU code (Alex Krizhevsky), took the convent and won ImageNet. That is when the current wave of "AI" started.
In summary, covnets and very loosely based on an ad hoc explanation to Hubel and Wiesel findings in primary visual cortex, which today in neuroscience are regarded as "incomplete" to say the least (more likely completely wrong). Now this stuff works to a degree, but really all these biological inspirations are very minimal.
The brain is a dynamic system and (some) neural networks are also dynamic systems, and a three layer neural network can learn to approximate any function. Thus, a neural network can approximate brain function arbitrarily well given time and space. Whether that simulation is conscious is another story. The Computational Cognitive Neuroscience Lab has been studying this topic for decades and has an online textbook here…
The first question is whether that neural network is learnable. Sure, some configuration of neurons may exist. Is it possible given enough time and space to discover what that configuration is, given a set of inputs and outputs?
The second question is whether "enough time and space" means "beyond the lifetime and resources of anyone alive," in which case it seems perfectly reasonable to me to call it a limitation. I generally want my software to work within my lifetime.
Earlier quoted context omitted.
I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing…
You seem to have replied on a tangent: how is what you describe not just "curve fitting"? Humans didn't magic that model up: you're ignoring the huge amount of human effort over thousands of years that it took to arrive at that model. If we gave a ML algorithm a similar amount of time and asked it to construct a simple model of the situation, it might very well hand back the formula you presented. Your entire post ba…
Alternatively, we could just run a bunch of experiments on data using ML models. Eventually, someone may have a wonderful idea and realize that we can just reduce the ML model into a parabola. Of course, this is due to intuition and not the ML model. Nevertheless, even though we end up at the same result, I contend the first result is different. It has a huge amount of information embedded into it about the assumptions we made into how the world works. When those assumptions are no longer satisfied, we have a rubric for constructing a fix. For example, if Galilean invariance no longer holds, we can fix the above model using the same sort of derivations to obtain relativistic expressions. Again, we could just throw more data at this new problem and fit an ML model to and perhaps someone would stare at this new model and realize that `E = m c^2`. However, I think that's discounting the embedded information in deriving these models and I don't think this information is present in ML models. ML models are generic. Our most powerful physical models are not.
Now, sure, once we have the models, we're just going to fit them to the data and it's all just curve fitting. Other fields call this parameter estimation, parameter identification, or a variety of other names. At that point it's all curve fitting. However, again, I contend the process for determining a new model is not.