Live data from Hacker News

Backpropagation is a leaky abstraction

medium.com

91–100 of 106 posts

Re: Backpropagation is a leaky abstraction

#91
post #71
post #3

The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).

Yes, after implementing a simple neural network in C (with AVX, and pthreads), "raw numpy" does sound funny! On the other hand, try implementing a convnet in numpy, especially the backprop, and you might start feeling some of its "rawness" :-)

As someone who also hand-coded a neural network implementation, forward and back prop as well as an RNN in C, yea "raw numpy" is a joke.

Something I've always hated about people that say "why do I have to write a backprop when TF does it for me?"

Here's why: you go to a company, they want you to incorporate machine learning into their c++ engine. Have fun using numpy, you said you knew machine learning right? implement backprop for me, you can do that right?

Re: Backpropagation is a leaky abstraction

#92

Earlier quoted context omitted.

> What is O(n log n)? Time complexity? And then worst case? Best case? Average case? Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?

And then worst case? Best case? Average case? I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however. Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or…

You can have Big-O of worst case and Big-O of average case. "Asymptotic complexity" itself does not imply worst case; the assumptions you make to construct the function itself determine worst case or average case.

Re: Backpropagation is a leaky abstraction

#93
post #48

Earlier quoted context omitted.

Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.

Because frameworks come and go. The important thing are the abstract concepts At that level, they assume you are pretty smart and capable of figuring out something like an API on your own time as needed. They'd rather you know what all these funky things in these APIs are doing at a core level so that you can employ them in an effective manner.

Letting the computer do the menial work of constructing formulas and generating code from user specifications is a fairly important framework feature and "abstract concept".

Re: Backpropagation is a leaky abstraction

#94
post #8

Backpropagation is a leaky abstraction in the sense that every algorithm/physics-principle/mathematical-theorem is a leaky abstraction. Take 'sort.' If you use a sort API for large N in a performance-critical section of your code without knowing if the implementation of that sort is an insertion-sort or a quicksort, "you would be nervous." Hence you are dealing with a leaky abstraction, per this article. I would much…

Performance is the same thing as correctness only in hard real time.

A better analogy might be stable sort versus unstable. If you need a stable sort, and unstable one is wrong.

(Not sure if that counts as an abstraction leak, though! It's just the semantics. Unstable sort doesn't leak that it is unstable; that's its behavior. If a sort is just documented as putting items in order, an actual implementation does leak information about whether or not it is stable.)

Re: Backpropagation is a leaky abstraction

#95

Earlier quoted context omitted.

Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.

Are you seriously asking that? Have you seen the example of learning multiplication vs using a calculator? What would you say in that case?

It's pretty much the same as looking at a multiplication table on a request basis.

Ironically, mere exposure to data is enouhg to learn from in ML, so why not here. Although, I'm not sure about the pedagogic aspect. I'd assume calculation by heart would be learned along the way, despite sending the initial message, it wasn't needed. Maybe starting slow is important, because it's that fundamental. But in hindsight, I really was good after half the elementary training.

I have a similar anecdote: I wasn't good at handwriting and always claimed I wouldn't need to. Now I don't need to, indeed, except for exams. But I actually have a hard time with caligraphy and that's a shame.

Re: Backpropagation is a leaky abstraction

#96
post #87

For anyone trying to learn backpropagation but having trouble with math, I can't recommend Matt Mazur's "Step by Step" guide [0] enough. What is great is that he is using real numbers so that one can check implementation for correctness. [0] https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

I used this recently when I was relearning NN theory (last time I looked at it before 2015 was the late 90's,) and I agree. It's a thorough walkthrough of the math that turned the lightbulbs back on enough for me to write a simple swift MLP NN without consulting other implementations.

Re: Backpropagation is a leaky abstraction

#97

> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?” How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"

Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.

There's at least two ways to go about learning ML, and I think ideally one should do both. One is to use a high level tool like TensorFlow or Azure ML Studio to experiment with what the tools can do with data. This can get you sufficient competency to use these algorithms in a practical way.

The other is to learn the foundations of those algorithms so you can best understand how to tune, apply and extend them. This is the path to mastery.

Re: Backpropagation is a leaky abstraction

#98
post #62

Earlier quoted context omitted.

In terms of big picture stuff, you're absolutely right that many DNNs are more "inspired by" the brain and less a faithful model of it. However, a lot of the things mentioned in your post are either overstated or outright wrong. For example: 1. Neurons, or more specifically, connections between neurons ("synapses"), absolutely do have weights, and the "strength" of synapses can be adjusted by a variety of properties…

This is exactly the kind of response I was hoping for. Thanks Matt! If it's not an inconvenience, could you drop any links to the topics you referenced, especially the ones that differed from what I've been studying, to charles@geuis.com? I'm going a bit deeper now and reading some studies from the early 90's and some that are more recent. It's kind of a crapshoot of what I can google for, so a guided search would be…

Hmmm...it's hard to do entire fields justice, but here's an attempt.

There are a couple of standard neurobiology textbooks, like Kandel, Jessel, and Schwartz's Principles of Neural Science, Purve et al.'s Neuroscience and Squire et al.'s Fundamental Neuroscience. These are huge books that cover a bit of everything, and you should know that they exist, but I wouldn't necessarily start there.

If you're specifically interested in computation, I would start with David Marr's Vision. It's quite old, but worth reading for the general approach he takes to problem-solving. He proposes attacking a problem along three lines: at the computational level ("what operations are performed?"), the algorithmic level ("how do we do those operations?"), and the implementation level ("how is the algorithm implemented").

From there, it depends on what you're interested in. At the single-cell level, Cristof Koch has a book called The Biophysics of Computation that "explains the repetoire of computational functions available to single neurons, showing how individual nerve cells can multiply, integrate, and delay synaptic input" (among other things). Michael London and Michael Häusser have a 2005 Annual Reviews in Neuroscience article about dendritic computation that hits on some similar themes (here: https://www.researchgate.net/publication/7712549_Dendritic_c... ), along with this short review (http://www.nature.com/neuro/journal/v3/n11s/full/nn1100_1171...) by Koch and Sergev, and a 2014 review by Brunel, Hakim, and Richardson (http://www.sciencedirect.com/science/article/pii/S0959438814...). Larry Abbott has also done interesting work in this space, as have Haim Sompolinsky and many others. Gordon Shepard and his colleagues maintain NEURON (a simulation package/platform) and a database of associated models (ModelDB) here: https://senselab.med.yale.edu/ if you want something to download and play with (they also do good original work themselves!)

Moving up a bit, the keywords for "weight adjustment" are something like synaptic plasticity, long-term potentiation/depression (LTP/LTD), and perhaps spike-timing dependent plasticity. The scholarpedia article on spike-timing dependant plasticity is pretty good (http://www.scholarpedia.org/article/Spike-timing_dependent_p... Scholarpedia is actually a pretty good resource for most of these topics. The intro books above will have pretty good treatments of this, though maybe not explicitly computational ones.

More to come, however I also just found this class from a bunch of heavy-hitters at NYU: http://www.cns.nyu.edu/~rinzel/CMNSF07/ Those papers are a good place to start!

Re: Backpropagation is a leaky abstraction

#99

Earlier quoted context omitted.

Are you seriously asking that? Have you seen the example of learning multiplication vs using a calculator? What would you say in that case?

It's pretty much the same as looking at a multiplication table on a request basis. Ironically, mere exposure to data is enouhg to learn from in ML, so why not here. Although, I'm not sure about the pedagogic aspect. I'd assume calculation by heart would be learned along the way, despite sending the initial message, it wasn't needed. Maybe starting slow is important, because it's that fundamental. But in hindsight, I…

> I have a similar anecdote: I wasn't good at handwriting and always claimed I wouldn't need to. Now I don't need to, indeed, except for exams. But I actually have a hard time with caligraphy and that's a shame.

Your anecdote is irrelevant because there is no _understanding_ to be gained by handwriting as opposed to typing.

I really can't be bothered with this conversation, sorry.

Re: Backpropagation is a leaky abstraction

#100
post #82

Earlier quoted context omitted.

And then worst case? Best case? Average case? I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however. Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or…

> I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however. No, asymptotic notation only refers to the behaviour of the function (or algorithm) as you take the limit N->inf, e.g. changes to the size of the input. But the nature of the input often changes the behavi…

Thanks for the correction!
Post reply on HN