31–40 of 52 posts
Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave?
I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
Least squares, gradient descent and linear regression separately? I get that he wants to point out the profundity and universality of the ideas encompassed in those techniques (& models; least squares and gradient descent are rightly thought of as (numerical) techniques, whereas the linear regression models is a, well, model) but that is like saying that arithmetic is fundamental to deep learning. Essentially, this "…
Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
Inspired by biology is typically a better way to think about it. Airplanes have wings inspired from biological birds, and they share some structural similarities, but in practice they serve very different functions.
https://en.wikipedia.org/wiki/Hilbert%27s_thirteenth_problem
Kolmogorov authored a paper titled "On Representation of Continuous Functions of Several Variables by Superpositions of Continuous Functions of Smaller Number of Variables," that basically solved this in 1961. This led to a nice back and forth series of papers between Kolmogorov and Arnold, but the one that becomes more important is Kolmogorov's paper, "On the Representation of Continuous Functions of Many Variables by Superposition of Continuous Functions of One Variable and Addition," in 1963. What this paper proves is that any continuous function defined on the n-dimensional unit cube can be represented by the superposition of 2n one dimensional continuous functions:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...
Now, the problem with this theorem is that it doesn't say how to find these magical 2n functions. However, in 1989 Cybbenko published the paper, "Approximation by Superpositions of Sigmoidal Functions," which both extends and weakens the above result. Basically, he loses the 2n bound, but gives a way to construct these functions by using a linear projection inside of a superposition of sigmoids. This led to the universal approximation theorem:
https://en.wikipedia.org/wiki/Universal_approximation_theore...
and I would contend the underpinnings for the modern neural net models. Now, is there any biology in there? No. It's a long series of function approximation papers. That said, I don't know the authors involved or what inspired them to write these papers. However, given that we have a documented history of dry function approximation papers that give us the mathematical power that we need to begin to justify these models, I tend to feel that the biological connections are oversold.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
A neuron is either activated or not and each of the many inputs can be either excitatory (encourages activation) or inhibitory (discourages activation). McCulloch and Pitts formalized this as a weighted average of the inputs that was then thresholded to 0 or 1. And they showed some basic theoretical results from that that gave it some credit as a model for how intelligence can arise from neurons. Essentially they said behavior can be described as a classifier.
AFAIK, they didn't go much into how the weights were actually learned. Different strategies were tried, but we ultimately started to soften the threshold function into the logistic function (to make the network differentiable) and solve for the weights by gradient descent.
Modern Deep Learning makes the additional assumption that neurons in the same layer are not interconnected. This assumption, along with the fact that we're just dealing with weighted averages, allows us to describe networks in matrix form, allows us to compute the gradients with backprop, and allows efficient simulation on the GPU. This assumption is more practical than biological.
> show me how it relates to how [...] humans will behave?
[This page][1] attempts to connect the dots between the McCulloch and Pitts model, the resulting classifiers, and behavior. Essentially, the theory was that neurons can be formalized into classifiers, and behavior is just the output of these classifiers. I don't know too much about modern neuroscience, but given the amazing results we are seen these days in vision, language, and planning, I'd say the central ideas of the theory are still credible.
[1]: http://www.mind.ilstu.edu/curriculum/modOverview.php?modGUI=...
Earlier quoted context omitted.
Geoff Hinton - surely. But I think most experts will disagree on the other two. In terms of deep fundamental contributions I don't the think other two have made much. I think Andrew Ng has been a great popularizer/marketing guy - primarily with that Cats project. Likewise Demis Hassabis has been a great application creator - with amazing results of course - AlphaGo, Atari, etc. On a side note: I lost all respect for…
From a pure technical point of view, I agree with you. But there is no doubt they all have played an important role in popularising deeplearning. I am fancisnated in the history of deeplearning and how it went from a field no one cared to what it is today.
It did 3 things:
1. Provided a usable solution for what was previously an intractable real world problem, large multiclass image classification, with decent accuracy
2. Crushed the prior benchmark on this task
3. Found a practical workaround to what was the biggest bottleneck, computation time, by utilizing GPUs (and made Nvidia stock explode /s)
The subsequent ImageNet competitions then later provided the perfect catalyst to refining and making deep neural nets mainstream. In parallel the sudden interest from everyone else who in turn started applying neural nets to pretty much every domain out there under the sun, was what I think ultimately made deep learning as it is to what it's today.
Earlier quoted context omitted.
Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.
Others have pointed out that matrix inversion is O(n^3) and hence computationally infeasible. It is also worth considering that the conditioning of X^t X, k(X^t X) can be as large as k(X)^2, so solving in this way can be very unstable.
Earlier quoted context omitted.
Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.
From a pedagogical point of view, I think it's a very strange choice to go with gradient descent in this case. It makes linear regression look like something much more complicated than it actually is. People might be misled into thinking they need to hand code gradient descent every time they do a regression, for their 100 point dataset.
Linear models are much older than computers, dating back to Gauss at least, and they do not have anything to do with gradient descent.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.