Backpropagation is a leaky abstraction • Hacker News Board

Re: Backpropagation is a leaky abstraction

#31

Post by smallnamespace » Tue, Dec 20, 2016, 1:05 AM UTC

quelltext wrote: post #28

fizixer wrote: post #26

Earlier quoted context omitted.

> If every major language provides an O(n log n) sort function, is it still a leaky abstraction? Yes, if the performance is still not acceptable and you look into the problem and discover that your scenario could benefit from radix sort, or one of partial sorting, or some kind of intermittent sorting, all of which would require investigating the specific case at a "white box" level, ignoring the existence of a black-…

The implementation is a black box but the performance characteristics could very well be part of the API contract. When you require and it's feasible to use radix sort you probably know not to use the standard sort function.

Yes, but reasoning about the API contract is exactly what being taught sorting algorithms helps you navigate, just like learning the guts of backprop helps the ML practitioner understand why things can go wrong.

Re: Backpropagation is a leaky abstraction

#32

Post by pekk » Tue, Dec 20, 2016, 1:06 AM UTC

cossatot wrote: post #19

cr0sh wrote: post #11

I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as…

I don't know a huge amount about neurology (or neural nets), but... Adult humans seem to learn faster when there is some combination of theory, examples, and experience (aka feedback). I'm a scientist and I have very little interest in neural nets for science because it contorts away the kind of equation-based systematics that we rely on to understand our world. The theory component is missing. I'm more interested in…

There are quite a few neurons in an adult human, and they aren't just in some kind of undifferentiated randomly connected neuron soup. So when we talk about things that adult humans do in terms of neurons, it is a bit like talking about Linux in terms of transistors. It is not that transistors are irrelevant to Linux, but...

Now replace the neurons with some tangentially related abstraction that is massively different from real neurons, well...

Re: Backpropagation is a leaky abstraction

#33

Post by alxmdev » Tue, Dec 20, 2016, 1:19 AM UTC

Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing.

Not to mention that computer science and software engineering are such young fields that it seems unhealthy to take readily available abstractions as absolute givens. Everything stands to improve, even products and concepts that have been around for decades and that everyone uses.

Re: Backpropagation is a leaky abstraction

#34

Post by pekk » Tue, Dec 20, 2016, 1:27 AM UTC

Retra wrote: post #16

cr0sh wrote: post #11

I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as…

>Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. This sounds like a (common) failure to understand how abstractions work. Bridges don't do calculus, but the bridge builder uses calculus to understand what bridges do use (the laws of nature), and thus the calculus abstraction is used to encode the behavior of bridges. Thus you can model bridges using calculus. Sim…

If you fit some data with a very flexible function approximator, that does not imply any kind of isomorphism between the function approximator itself and the process generating the data.

Some people cannot understand this, and believe that if you can closely fit the output of a process with a neural network that it implies the process itself is in some way related to neural networks.

Re: Backpropagation is a leaky abstraction

#35

Post by stuxnet79 » Tue, Dec 20, 2016, 1:34 AM UTC

Retra wrote: post #16

cr0sh wrote: post #11

I do think the complaint on having to write the backward pass seems especially shallow; finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations). IIRC, when I took the ML Class in 2011, we used Octave, but Ng had us first write stuff "the hard way" - so we'd understand what was going on later when we used Octave's methods. Something about this article as…

>Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. This sounds like a (common) failure to understand how abstractions work. Bridges don't do calculus, but the bridge builder uses calculus to understand what bridges do use (the laws of nature), and thus the calculus abstraction is used to encode the behavior of bridges. Thus you can model bridges using calculus. Sim…

Very well said. It took me a long time to "get it" but once I crossed that threshold I started viewing different mathematical techniques as hammers, some more suited to modeling certain phenomena than others.

Re: Backpropagation is a leaky abstraction

#36

Post by posterboy » Tue, Dec 20, 2016, 1:34 AM UTC

fizixer wrote: post #8

Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…

> "because you registered for this class."

The Authoritative Argument is not very convincing when your good authority over the course material being challenged. Maybe there is some irony or deeper truth, lost on me. Are you perhaps a leaky abstraction?

edit:

> The "code reuse as much as possible" mantra applies to when ...

... you are able to reasonably compromise, got it.

Re: Backpropagation is a leaky abstraction

#37

Post by posterboy » Tue, Dec 20, 2016, 1:51 AM UTC

rahrahrah wrote: post #29

> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?” How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"

Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.

Re: Backpropagation is a leaky abstraction

#38

Post by posterboy » Tue, Dec 20, 2016, 1:57 AM UTC

alxmdev wrote: post #33

Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing. Not to mention…

> There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end

Yes there is, watch someone else do it. In fact there are three ways to learn, as the saying goes: trial and error, copying, and insight. I'd be hard pressed to explain the difference of trial and error vs insight, but I wouldn't confuse them either, because only one of them is painful.

Re: Backpropagation is a leaky abstraction

#39

Post by sn9 » Tue, Dec 20, 2016, 2:07 AM UTC

posterboy wrote: post #38

alxmdev wrote: post #33

Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing. Not to mention…

> There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end Yes there is, watch someone else do it. In fact there are three ways to learn, as the saying goes: trial and error, copying, and insight. I'd be hard pressed to explain the difference of trial and error vs insight, but I wouldn't confuse them either, because only one of them is painful.

Watching someone else do it is still less effective than bashing your head against the problem yourself before watching them do it.

Whether or not you succeed in solving it on your own, it will emotionally invest you in the problem and its solution while showing you what didn't work and having a better handle on the shape of the problem. This lets you get more out of seeing someone else work out the solution.

Re: Backpropagation is a leaky abstraction

#40

Post by alxmdev » Tue, Dec 20, 2016, 2:16 AM UTC

posterboy wrote: post #38

alxmdev wrote: post #33

Questions like "Why do we have to write X, when framework Y does it for you?" are why I dislike the reinventing the wheel analogy, especially when it finds its way in education. There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end. Students complaining about implementing a foundational algorithm instead of using a framework is depressing. Not to mention…

> There's no substitute for the deep understanding you get by solving a complex problem yourself from beginning to end Yes there is, watch someone else do it. In fact there are three ways to learn, as the saying goes: trial and error, copying, and insight. I'd be hard pressed to explain the difference of trial and error vs insight, but I wouldn't confuse them either, because only one of them is painful.

I think there are intangible things that get lost when you watch someone else do something as opposed to doing it yourself. The exercise is about going through the mental motions of understanding the problem, designing a solution, and iterating on it until it's correct. The last part is all about learning from your own mistakes, seeing what specific things trip you up, so you know to improve on them. That's not something you can get by watching others.

I agree that there isn't enough time in a life to learn everything you'd want to first hand or from a low level of abstraction, but school should be a place to do as much of it as possible. Just my 2c.