Backprop is just automatic differentiation. The end.
[deleted]
Backpropagation is a leaky abstraction
21–30 of 106 posts
Re: Backpropagation is a leaky abstraction
#22Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…
> You can use it without worrying much about the details.
I invite you to start sorting data which is often already sorted (or often all identical) and tell me you didn't need to worry about the details.
Re: Backpropagation is a leaky abstraction
#23Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
If someone wants to know a little calculus for a small area of engineering, do they really need to learn various manual techniques for calculating integrals? Why can't they use Mathematica? (What if they're just taking calculus as a requirement of CS or premed?) They're not planning on developing new integrations techniques. Does a pilot need to learn fluid dynamics like an airplane engineer?
Maybe educational students should only require what's actually practical. That can include things that can be solved by computers if they teach ideas that have practical value.
Re: Backpropagation is a leaky abstraction
#24Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…
Speaking from experience, I've had to worry about what implementation of sort is being used in many languages, from Java to C++, even Go and Python.
There are a lot of details to get right: how are elements compared? Is the sort stable? Is it efficient for small N? Is it efficient for nearly-sorted arrays? Is it efficient when almost all the elements compare equal? Is it guaranteed O(n log n) or average? If average, is there an input that reliably triggers n^2 behavior making it a DDoS vector?
Anything is a leaky abstraction when you care enough.
Re: Backpropagation is a leaky abstraction
#25Backprop is just automatic differentiation. The end.
That's a succinct way of putting it.
Re: Backpropagation is a leaky abstraction
#26Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…
> If every major language provides an O(n log n) sort function, is it still a leaky abstraction?
Yes, if the performance is still not acceptable and you look into the problem and discover that your scenario could benefit from radix sort, or one of partial sorting, or some kind of intermittent sorting, all of which would require investigating the specific case at a "white box" level, ignoring the existence of a black-box O(n-log-n) sort API call.
Re: Backpropagation is a leaky abstraction
#27Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
If someone wants to know a little calculus for a small area of engineering, do they really need to learn various manual techniques for calculating integrals? Why can't they use Mathematica? (What if they're just taking calculus as a requirement of CS or premed?) They're not planning on developing new integrations techniques. Does a pilot need to learn fluid dynamics like an airplane engineer? Maybe educational studen…
> If someone wants to know a little calculus for a small area of engineering, do they really need to learn various manual techniques for calculating integrals?
Yes, but the logic goes the other way: if they are unable to use the technique manually, then one may conclude they do not understand the idea well enough to use it, whether or not Mathematica is available. The two, manual application and understanding/intuition, go hand in hand and can't be separated. My understanding is that this has even been studied more formally, there is a paper I remember reading by Kahneman and somebody else [1] about the development of intuition and the tension between heuristics-and-biases and naturalistic decision making. In short, don't trust people so much, if they say they understand something but are unable to actually do it, be a bit more skeptical.
[1] Found it: https://www.ncbi.nlm.nih.gov/pubmed/19739881 On second thought, it's probably not as relevant as I remembered it.
Re: Backpropagation is a leaky abstraction
#28Earlier quoted context omitted.
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…
> If every major language provides an O(n log n) sort function, is it still a leaky abstraction? Yes, if the performance is still not acceptable and you look into the problem and discover that your scenario could benefit from radix sort, or one of partial sorting, or some kind of intermittent sorting, all of which would require investigating the specific case at a "white box" level, ignoring the existence of a black-…
The implementation is a black box but the performance characteristics could very well be part of the API contract. When you require and it's feasible to use radix sort you probably know not to use the standard sort function.
Re: Backpropagation is a leaky abstraction
#29> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?”
How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"
Re: Backpropagation is a leaky abstraction
#30I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as…
I think everyone agrees that ANN are a useful abstraction of what happens in real neural networks. A "closer" abstraction in some sense, as you mention, is what is being called Spiking Neural Networks, so you may want to read up on that if you're interested. I don't think they are strictly more powerful in any way, for what it's worth, as they more or less just trade continuous-domain discrete-time computations for d…
ANNs would be useful or not completely apart from the relationship to anything "neural." I often wish that the association had never been drawn, it just doesn't seem very useful and it is always misleading so many people.