41–50 of 52 posts
The article mentions that GPUs are on average 50-200 times faster for deep learning, I’m curious on how he came to that number. It has a lot to do with the code and the frameworks used. I haven’t come across a good comparison, most figures seems to be taken out of the blue.
Edit: There we go with the downvotes, I knew it that deep learning guys can't stand this claim (but it's true, as the post itself goes to show in great length... :-))
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
We say ANNs are "based on how the brain works" because the original mathematical model was an attempt by McCulloch and Pitts to explain how complex behavior arises from networks of simple neurons. A neuron is either activated or not and each of the many inputs can be either excitatory (encourages activation) or inhibitory (discourages activation). McCulloch and Pitts formalized this as a weighted average of the input…
Calling those chained regressions similar to the brain is about as correct as saying that a 3y old's drawing of a car is similar to a real Tesla...
Earlier quoted context omitted.
This is something I've also struggled with. I find it hard to read deep learning papers because I need to translate each math notation, thus struggling to get the bigger picture. I'm fond of the bottom-up approach, e.g. I started by mastering C and wrote my own libraries. But for deep learning I lean towards the opposite, starting with high-level libraries. When I want to understand the theory I search for simple pyt…
"Machine Learning Engineer" is a title we're going to see more and more of (and we're already seeing a lot). Its one thing to know the math and theory to design, train, and tune the algorithm your company needs. But implementing it into production, at scale? That's not the same person. Ideally, you have Person/Team A, who designs but knows enough about implementation to keep that in mind during their process, and Per…
So the compromise is usually as you describe, but you bear the cost of translation issues no matter how you do this. It's worth remembering that it is a compromise.
I think systems like tensorflow are implicitly a recognition of this, allowing lower impedance between the groups.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.
Earlier quoted context omitted.
Inspired by biology is typically a better way to think about it. Airplanes have wings inspired from biological birds, and they share some structural similarities, but in practice they serve very different functions.
I would even say that this is somewhat revisionist history. From my perspective, this all started from an attempt by Kolmogorov to solve Hilbert's 13th problem: https://en.wikipedia.org/wiki/Hilbert%27s_thirteenth_problem Kolmogorov authored a paper titled "On Representation of Continuous Functions of Several Variables by Superpositions of Continuous Functions of Smaller Number of Variables," that basically solved th…
Earlier quoted context omitted.
I would even say that this is somewhat revisionist history. From my perspective, this all started from an attempt by Kolmogorov to solve Hilbert's 13th problem: https://en.wikipedia.org/wiki/Hilbert%27s_thirteenth_problem Kolmogorov authored a paper titled "On Representation of Continuous Functions of Several Variables by Superpositions of Continuous Functions of Smaller Number of Variables," that basically solved th…
That timeline seems to miss that Yann LeCun was already working on ConvNets in 1988. I don't think anyone waited for the Universal Approximation theorem to start building neural architectures, it was just a tangentially interesting mathematical result.
http://yann.lecun.com/exdb/publis/index.html
More generally, a common trope in NN papers and books is to draw a graph for matrix-vector multiplication and then draw the analogy that these are like neurons in the brain and this represents their connectivity. This is an example of the kind of backwalking biological analogies that frustrate me. Again, certainly, I don't know the motivations behind everyone in the field, but I do contend that many of the more powerful theorems have nothing to do with biology and have other origins.
Earlier quoted context omitted.
We say ANNs are "based on how the brain works" because the original mathematical model was an attempt by McCulloch and Pitts to explain how complex behavior arises from networks of simple neurons. A neuron is either activated or not and each of the many inputs can be either excitatory (encourages activation) or inhibitory (discourages activation). McCulloch and Pitts formalized this as a weighted average of the input…
First of all, neurons don't have just one activation function. Each dendrite has. So, anything from dozens to thousands. Second, that definition doesn't cover the entire issue of multiple feedback loops. Third, this doesn't cover memory effects at structural (cytoskeleton) and local levels (vesicles), much less generic levels (RNA and your genes). And then we haven't even gotten into metabolomic and epigenetic wririn…
McCulloch and Pitts published in the 1950s. Of course we know more about the brain now.
If I were to ask you "How does intelligence arise from a network of activations?" Would you genuinely say that it has nothing to do with the McCulloch and Pitts theory?
Earlier quoted context omitted.
First of all, neurons don't have just one activation function. Each dendrite has. So, anything from dozens to thousands. Second, that definition doesn't cover the entire issue of multiple feedback loops. Third, this doesn't cover memory effects at structural (cytoskeleton) and local levels (vesicles), much less generic levels (RNA and your genes). And then we haven't even gotten into metabolomic and epigenetic wririn…
I mean, doi. McCulloch and Pitts published in the 1950s. Of course we know more about the brain now. If I were to ask you "How does intelligence arise from a network of activations?" Would you genuinely say that it has nothing to do with the McCulloch and Pitts theory?
But more generally, I am just so tied of this "brain metaphor" on deep learning. It is a funny way to wake up your students (well, at least 10 years ago it was...), but trying to stretch this metaphor much more than that is just painful. Heck, even the activating "functions" (plural, as we now know) in a neutron isn't really a set (!) of (singular, independent) functions, it's just a top level name for a mind-boggling number of things happening as neurons "fire", with a mathematical formalism to approximate what's going on. In fact, calling an activation a "function" is probably belittling the biological processes behind them.
I know it is popular to say that these techniques are based on how the brain works, but when I read about them, I have my doubts. Can anyone take a real world example of human behavior and show me how it relates to how these techniques predict humans will behave? I love the field but feel like there is a temptation to take giant leaps not supported by other observations.