Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backpropagation is a leaky abstraction

medium.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Backpropagation is a leaky abstraction

#81

Post by 4ad »

im3w1l wrote: post #79
sirclueless wrote: post #24

Earlier quoted context omitted.

Speaking from experience, I've had to worry about what implementation of sort is being used in many languages, from Java to C++, even Go and Python. There are a lot of details to get right: how are elements compared? Is the sort stable? Is it efficient for small N? Is it efficient for nearly-sorted arrays? Is it efficient when almost all the elements compare equal? Is it guaranteed O(n log n) or average? If average,…

Those properties should ideally be part of the documentation so that the abstraction stops leaking.

One could argue that if these details are necessary for correct/performant operation, then it is a leaky abstraction.

Re: Backpropagation is a leaky abstraction

#82

Post by 4ad »

dkersten wrote: post #76
masklinn wrote: post #70

Earlier quoted context omitted.

> What is O(n log n)? Time complexity? And then worst case? Best case? Average case? Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?

And then worst case? Best case? Average case? I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however. Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or…

> I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however.

No, asymptotic notation only refers to the behaviour of the function (or algorithm) as you take the limit N->inf, e.g. changes to the size of the input. But the nature of the input often changes the behavior as well.

Re: Backpropagation is a leaky abstraction

#83

Post by ebalit »

geuis wrote: post #15

Edit: hmm not sure why this comment is getting downvoted. Backprop isn't just a leaky abstraction. There's not really any evidence yet that biological neural networks use anything like backprop. So its important that students be taught the low-level aspects of the current state of the art so that better architectures can be invented in the future. (Note: at the very end of this comment, I am leaving a link to one hyp…

If you're interested in Reinforcement Learning and Spiking Neural Network, you should look into Izhikevich work on dopamine-modulated STDP:

http://www.izhikevich.org/publications/dastdp.pdf

Re: Backpropagation is a leaky abstraction

#84

Post by uryga »

mattkrause wrote: post #42
geuis wrote: post #15

Edit: hmm not sure why this comment is getting downvoted. Backprop isn't just a leaky abstraction. There's not really any evidence yet that biological neural networks use anything like backprop. So its important that students be taught the low-level aspects of the current state of the art so that better architectures can be invented in the future. (Note: at the very end of this comment, I am leaving a link to one hyp…

In terms of big picture stuff, you're absolutely right that many DNNs are more "inspired by" the brain and less a faithful model of it. However, a lot of the things mentioned in your post are either overstated or outright wrong. For example: 1. Neurons, or more specifically, connections between neurons ("synapses"), absolutely do have weights, and the "strength" of synapses can be adjusted by a variety of properties…

@geouis asked you to email him some reading material on this in a sibling comment. if you do send them something, could you post it here too?

Re: Backpropagation is a leaky abstraction

#85

Post by ezanmoto »

sirclueless wrote: post #24
skybrian wrote: post #18

Earlier quoted context omitted.

I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…

Speaking from experience, I've had to worry about what implementation of sort is being used in many languages, from Java to C++, even Go and Python. There are a lot of details to get right: how are elements compared? Is the sort stable? Is it efficient for small N? Is it efficient for nearly-sorted arrays? Is it efficient when almost all the elements compare equal? Is it guaranteed O(n log n) or average? If average,…

I feel that if you care that much about the internals of the abstraction then you don't necessarily want an abstraction anymore, but rather the concrete thing that is being abstracted over. I think abstraction is an admirable quality that has numerous advantages, but if you need guarantees on the properties of what's under the abstraction then an abstraction is probably not what's needed at that point.

Re: Backpropagation is a leaky abstraction

#86

Post by im3w1l »

4ad wrote: post #81
im3w1l wrote: post #79

Earlier quoted context omitted.

Those properties should ideally be part of the documentation so that the abstraction stops leaking.

One could argue that if these details are necessary for correct/performant operation, then it is a leaky abstraction.

To me a leaky abstraction is an abstraction that does not expose all the relevant details. So if those details are written on the spec sheet of the black box then there is no leak.

If the box is only labelled with O(nlog n) without specifying constants then there is a leak.

Re: Backpropagation is a leaky abstraction

#87

Post by annnnd »

For anyone trying to learn backpropagation but having trouble with math, I can't recommend Matt Mazur's "Step by Step" guide [0] enough. What is great is that he is using real numbers so that one can check implementation for correctness.

[0] https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

Re: Backpropagation is a leaky abstraction

#88

Post by smaddox »

geuis wrote: post #15

Edit: hmm not sure why this comment is getting downvoted. Backprop isn't just a leaky abstraction. There's not really any evidence yet that biological neural networks use anything like backprop. So its important that students be taught the low-level aspects of the current state of the art so that better architectures can be invented in the future. (Note: at the very end of this comment, I am leaving a link to one hyp…

One hypothesis is that the neural units in a CNN model hundreds or thousands of individual neurons, including some neurons used to produce/transmit error signals.

This seems reasonable, but I doubt there is any single simple and general abstraction that can describe the learning algorithms used by physical neurons. It seems more likely that the brain uses many highly specialized algorithms for different regions of the brain, each shaped by a ~billion years of evolution.

Re: Backpropagation is a leaky abstraction

#89

Post by annnnd »

sherjilozair wrote: post #59
fizixer wrote: post #8

Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…

I think Andrej is arguing over and above the reasons you cite. Not only should you learn backprop because of the same reason you learn to do 2+2, but you should learn backprop ALSO because it's a leaky abstraction. This is a non-trivial statement, because there are other things which are not leaky. For example, he's not arguing that deep learning practitioners should also learn assembly programming or go into how CUB…

> Backprop knowledge, however, is much more crucial to design novel deep learning systems.

I would argue that it's not just for that. You need to understand what is happening inside DNN if you want to construct it properly.

Post reply on HN