“Deep Learning has outlived its usefulness as a buzz-phrase” • Hacker News Board

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#91

Post by albertzeyer » Sat, Jan 06, 2018, 2:37 PM UTC

BucketSort wrote: post #19

I believe this paper by Marcus ( https://arxiv.org/ftp/arxiv/papers/1801/1801.00631.pdf ) earlier this week inspired this. Edit: I don't mean Marcus inspired the term differentiable programming; he inspired LeCun to emphasize the wider scope of deep learning after Marcus attacked it. In fact, LeCun liked a post on twitter rebutting Marcus' paper that also talks about differentiable programming: https://twitter.com/td…

Discussion about that paper here: https://news.ycombinator.com/item?id=16083469

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#92

Post by tmyklebu » Sat, Jan 06, 2018, 3:53 PM UTC

mturmon wrote: post #65

bra-ket wrote: post #35

Earlier quoted context omitted.

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

Following a gradient is smarter than trial and error. You can make an argument that, in high-dimensional parameter spaces, it’s hard to do better (because, gradient descent is linear in the number of dimensions). Ordinary Metropolis-Hastings, for example, is closer to trial and error.

> You can make an argument that, in high-dimensional parameter spaces, it’s hard to do better (because, gradient descent is linear in the number of dimensions).

Nitpick: If you're interested in solving optimisation problems, it's very easy to do better than gradient descent. Gradient descent performs very poorly when the directional curvature of the objective function varies too much with the direction. Newton's method, quasi-Newton methods, and nonlinear conjugate gradient are some of the more ingenious, beautiful, and clean ways; there are also some dirty, hacky ways to go.

It is a little bit interesting that fancier optimisation algorithms than gradient descent are unnecessary or unhelpful in some large-scale applications.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#93

Post by wadkar » Sat, Jan 06, 2018, 5:26 PM UTC

tedivm wrote: post #33

kinkrtyavimoodh wrote: post #18

Earlier quoted context omitted.

I opened it in an Incognito window and it worked fine. Public posts on FB can be seen without logging in.

How long did you wait on the page? At first when it loads you can read it just fine, but then an obnoxious pop up appears. There is a little "not now" link on the bottom that will get you back to the post, but I can see how someone would miss that (it's a great example of a bad UI).

> it’s a great example of a bad UI

I think it’s a good example of UX that encourages visitor to sign up - which is why it exists. A part of what makes it good is precisely what you mentioned: very easy to miss the button

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#94

Post by dang » Sat, Jan 06, 2018, 6:14 PM UTC

tomdre wrote: post #15

Off topic. Yann LeCun really looks like Michael Moore who looks like Peter Griffin.

Please don't post unsubstantive comments here.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#95

Post by mturmon » Sat, Jan 06, 2018, 6:25 PM UTC

tmyklebu wrote: post #92

mturmon wrote: post #65

Earlier quoted context omitted.

Following a gradient is smarter than trial and error. You can make an argument that, in high-dimensional parameter spaces, it’s hard to do better (because, gradient descent is linear in the number of dimensions). Ordinary Metropolis-Hastings, for example, is closer to trial and error.

> You can make an argument that, in high-dimensional parameter spaces, it’s hard to do better (because, gradient descent is linear in the number of dimensions). Nitpick: If you're interested in solving optimisation problems, it's very easy to do better than gradient descent. Gradient descent performs very poorly when the directional curvature of the objective function varies too much with the direction. Newton's meth…

You mention Newton’s method, but of course that requires second order information which, as I mentioned, is not generally workable in high dimensions. You have to be careful with quasi-Newton methods like conjugate gradient for the same reason.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#96

Post by cs702 » Sat, Jan 06, 2018, 6:34 PM UTC

sitkack wrote: post #79

cs702 wrote: post #49

I wish we could come up with a catchier name, but I LOVE the idea of calling this programming , because that is precisely what we do when we compose deep neural nets. For example, here's how you compose a neural net consisting of two "dense" layers (linear transformations), using Keras's functional API, and then apply these two layers to some tensor x to obtain a tensor y: f = Dense(n) g = Dense(n) y = f(g(x)) This l…

NN are _just_ transfer functions. Look up tables. Or really dense maps. So f(g(x)) make total sense. But I dont think these are the interesting combinations. I think giving one NN the training experience of another, plus the feedback on "correct inference" will be when on NN trains its replacement.

Yes, of course. f(g(x)) was the simplest possible example I could come up with to illustrate the point :-)

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#97

Post by tmyklebu » Sat, Jan 06, 2018, 6:45 PM UTC

mturmon wrote: post #95

tmyklebu wrote: post #92

Earlier quoted context omitted.

> You can make an argument that, in high-dimensional parameter spaces, it’s hard to do better (because, gradient descent is linear in the number of dimensions). Nitpick: If you're interested in solving optimisation problems, it's very easy to do better than gradient descent. Gradient descent performs very poorly when the directional curvature of the objective function varies too much with the direction. Newton's meth…

You mention Newton’s method, but of course that requires second order information which, as I mentioned, is not generally workable in high dimensions. You have to be careful with quasi-Newton methods like conjugate gradient for the same reason.

> You mention Newton’s method, but of course that requires second order information which, as I mentioned, is not generally workable in high dimensions.

Why would you say that second-order information is "not generally workable in high dimensions"? We regularly run Newton's method on problems with tens of millions of variables today.

And Newton's method isn't the only way to use second-order information. It is easy to access, for example, Hessian-times-vector information using the same reverse-mode differentiation that's so popular today, using only a constant factor more time.

> You have to be careful with quasi-Newton methods like conjugate gradient for the same reason.

What reason, exactly, is that?

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#98

Post by YeGoblynQueenne » Sat, Jan 06, 2018, 8:30 PM UTC

kinkrtyavimoodh wrote: post #20

ehsankia wrote: post #5

1. Differentiable Programming is horrible branding. It's hard to say, not catchy, and not as easily decipherable. 2. Isn't the evolution of Deep Networks more advance setups such as GANs, RNNs, and so on?

> Differentiable Programming is horrible branding. It's hard to say, not catchy, and not as easily decipherable Tell that to the people who deliberately popularized the term Dynamic Programming for something that was neither dynamic nor programming. ____ (From Wiki) Bellman explains the reasoning behind the term dynamic programming in his autobiography, Eye of the Hurricane: An Autobiography (1984, page 159). He expl…

>> Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible.

Now I have to try:

  "Dynamic, multimodal failure" (fail).

  "Dynamic instigation of pain for information retrieval" (torture).

  "Dynamic evisceration of underage humans" (slaughtering of children).

  "Dynamic destruction of useful resources" (environment destruction).

  "An algorithm for calculating dynamic stool-rotor collision physics" (shit hits the fan).

Not terribly good I guess but I think not that bad either.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#99

Post by rspeer » Sun, Jan 07, 2018, 1:12 AM UTC

clarry wrote: post #90

contrarian_ wrote: post #86

Earlier quoted context omitted.

> artificial neurons Differentiable functions

Basically you're implying binary neurons, neuroevolution (which works on non-differentiable functions) etc. aren't a thing. Or at least they're not (working with) neural nets. It's almost like SGD just made decades of AI research into neural networks just vanish.

I don't see that implication at all.

Nobody is claiming that the definition of "differentiable programming" should be identical to the definition of "neural net". The claim is, if you want to assign a name to the thing that TensorFlow, PyTorch, and similar frameworks do, it's "differentiable programming".

If you want to make a non-differentiable neural net, knock yourself out. The research still exists and nobody is stopping you.

But while we're talking about terminology, I'd encourage you to stop referring to the units as "neurons". The false analogy to biology just confuses people.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#100

Post by qwerty456127 » Sun, Jan 07, 2018, 4:37 AM UTC

bufferoverflow wrote: post #83

qwerty456127 wrote: post #68

Earlier quoted context omitted.

No. E.g. I'm fine with Telegram & Signal + WhatsApp & Skype for the elderly. Having no intention to become a public person, do I really need a public profile on a web site that's whole purpose is to spy on me everywhere, analyse my behavior and contacts, sell the data to others and show me ads?

I was recently testing video quality of video chats on a relatively shitty connection, and somehow Skype came out way ahead of Whatsapp and Hangouts.

I absolutely believe you. When I actually need to voice-call somebody (e.g. mom, granny or a client company CEO, all the other people I meet prefer texting) I use Skype - it does this much better than any of the competitors. So it sounds fairly probable it does video better than others too.