Backpropagation is a leaky abstraction
medium.com
Backpropagation is a leaky abstraction
1–10 of 106 posts
Re: Backpropagation is a leaky abstraction
#2People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.
Re: Backpropagation is a leaky abstraction
#3The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).
Re: Backpropagation is a leaky abstraction
#4People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.
Hear, hear. Indeed, "welcome" to everybody who takes the time to understand what (s)he is talking about.
Re: Backpropagation is a leaky abstraction
#5People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.
> People complaining about having to implement backprop in a ML class?
Wouldn't that suggest that other classes the students take have not been academic enough - i.e. that they focus too much on "things you might use day-to-day" vs "this is how/why things work"?
Re: Backpropagation is a leaky abstraction
#6People complaining about having to implement backprop in a ML class? Nice one. Had to hand in this exact assignment last Friday. A valuable lesson, that's for sure. Generally I love how a proper lesson on Machine Learning really contradicts the "universal power weapon" narrative media put on machine learning in the last few years. Everything is not so magic anymore when it boils down to proper algorithm use.
> People complaining about having to implement backprop in a ML class? Wouldn't that suggest that other classes the students take have not been academic enough - i.e. that they focus too much on "things you might use day-to-day" vs "this is how/why things work"?
This attitude is not uncommon in programming / CS. It's similar to those who pine about learning a half dozen search algorithms when they rarely need to implement them in practice.
Re: Backpropagation is a leaky abstraction
#7> Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?
Why do you have to learn to calculate integrals and derivatives in school, or how compilers work internally? Same answer. But seriously, the CS231 class is excellent, and Andrej is an excellent teacher. You can follow along at home (which is what I am doing.) The syllabus (at http://cs231n.stanford.edu/syllabus.html) has the course notes and the assignments. The assignments are self grading, you know when you have it programmed correctly. The lectures are here: https://www.youtube.com/playlist?list=PLlJy-eBtNFt6EuMxFYRiN...
Re: Backpropagation is a leaky abstraction
#8Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article.
I would much rather respond to the students who ask about why they need to know the details, by saying "because you registered for this class."
Essentially the question "if you’re never going to write backward passes once the class is over, why practice writing them?" is another way of asking "why reinvent the wheel?". And the best answer is "because learning is all about reinventing the wheel". The "code reuse as much as possible" mantra applies to when you're "using" a technique to do something else, not when you're "learning" the technique itself. They might as well register for Calculus and ask "why learn integrals and derivatives when mathematica can do them for you", or take an Aerodynamics class and ask "why learn fluid mechanics and dynamics, heck newton's laws, when an airplane can run on autopilot." I doubt "because calculus is a leaky abstraction" or "because fluid dynamics is a leaky abstraction" is a good answer to that.
Re: Backpropagation is a leaky abstraction
#9Backprop is just automatic differentiation. The end.
Re: Backpropagation is a leaky abstraction
#10Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…
"because fluid dynamics is a leaky abstraction"
No pun intended!