“Deep Learning has outlived its usefulness as a buzz-phrase” • Hacker News Board

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#71

Post by nostrademons » Sat, Jan 06, 2018, 5:45 AM UTC

bra-ket wrote: post #35

rememberlenny wrote: post #3

[Text from post] OK, Deep Learning has outlived its usefulness as a buzz-phrase. Deep Learning est mort. Vive Differentiable Programming! Yeah, Differentiable Programming is little more than a rebranding of the modern collection Deep Learning techniques, the same way Deep Learning was a rebranding of the modern incarnations of neural nets with more than two layers. But the important point is that people are now build…

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

As much of a pity as that after 3 billion years, all life is still based upon selection bias and random genetic mutations, a brute force trial and error?

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#72

Post by outlace » Sat, Jan 06, 2018, 6:07 AM UTC

But differentiable programming would exclude deep neural nets trained by evolutionary methods/genetic algorithms since those are gradient free. With the term deep learning I think the focus is correctly on the “deep” (compositional) nature of these models and not necessarily the training algorithm, of which there are many.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#73

Post by electricslpnsld » Sat, Jan 06, 2018, 6:12 AM UTC

bra-ket wrote: post #35

rememberlenny wrote: post #3

[Text from post] OK, Deep Learning has outlived its usefulness as a buzz-phrase. Deep Learning est mort. Vive Differentiable Programming! Yeah, Differentiable Programming is little more than a rebranding of the modern collection Deep Learning techniques, the same way Deep Learning was a rebranding of the modern incarnations of neural nets with more than two layers. But the important point is that people are now build…

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

If you can’t reasonably get at or use second order information, how else are you going to optimize arbitrary objectives?

Well, come to think of it it, why don’t DL approaches use BFGS instead of gradient descent?

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#74

Post by jph00 » Sat, Jan 06, 2018, 6:18 AM UTC

electricslpnsld wrote: post #73

bra-ket wrote: post #35

Earlier quoted context omitted.

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

If you can’t reasonably get at or use second order information, how else are you going to optimize arbitrary objectives? Well, come to think of it it, why don’t DL approaches use BFGS instead of gradient descent?

Second order methods are attracted to saddle points in high dimensional spaces. The math and practice of optimizing these surfaces has a lot of nuances like this so much of the stuff you learn in your convex optimization class doesn't apply too well.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#75

Post by fwilliams » Sat, Jan 06, 2018, 6:39 AM UTC

electricslpnsld wrote: post #73

bra-ket wrote: post #35

Earlier quoted context omitted.

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

If you can’t reasonably get at or use second order information, how else are you going to optimize arbitrary objectives? Well, come to think of it it, why don’t DL approaches use BFGS instead of gradient descent?

There is literature on Quasi-Newton and Krylov Subspace methods for training Neural Networks. For example, https://dl.acm.org/citation.cfm?id=3104516.

I think the primary reason that such methods are not used much in practice is memory and computational cost: each function evaluation is expensive and you need to solve a very large system at every iteration.

Also to reply to a sibling comment, you can add momentum and step length adjustments to second-order methods in much the same way as in steepest-descent to help escape saddles. The only difference is how the descent direction is chosen for the optimization.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#76

Post by Ormus » Sat, Jan 06, 2018, 6:52 AM UTC

BucketSort wrote: post #19

I believe this paper by Marcus ( https://arxiv.org/ftp/arxiv/papers/1801/1801.00631.pdf ) earlier this week inspired this. Edit: I don't mean Marcus inspired the term differentiable programming; he inspired LeCun to emphasize the wider scope of deep learning after Marcus attacked it. In fact, LeCun liked a post on twitter rebutting Marcus' paper that also talks about differentiable programming: https://twitter.com/td…

I can guarantee you that Marcus has in no way ever inspired LeCun.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#77

Post by letlambda » Sat, Jan 06, 2018, 6:59 AM UTC

bmc7505 wrote: post #52

cs702 wrote: post #49

I wish we could come up with a catchier name, but I LOVE the idea of calling this programming , because that is precisely what we do when we compose deep neural nets. For example, here's how you compose a neural net consisting of two "dense" layers (linear transformations), using Keras's functional API, and then apply these two layers to some tensor x to obtain a tensor y: f = Dense(n) g = Dense(n) y = f(g(x)) This l…

I hate the name, but LOVE the idea of calling this programming... What would you call it instead?

∇programming

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#78

Post by plafl » Sat, Jan 06, 2018, 7:15 AM UTC

electricslpnsld wrote: post #73

bra-ket wrote: post #35

Earlier quoted context omitted.

it's really a pity that after 75 years of AI research the best thing we've got is still based on gradient descent, a brute force trial and error.

If you can’t reasonably get at or use second order information, how else are you going to optimize arbitrary objectives? Well, come to think of it it, why don’t DL approaches use BFGS instead of gradient descent?

I have not used it but there is an implementation in PyTorch: http://pytorch.org/docs/master/optim.html#torch.optim.LBFGS

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#79

Post by sitkack » Sat, Jan 06, 2018, 7:36 AM UTC

cs702 wrote: post #49

I wish we could come up with a catchier name, but I LOVE the idea of calling this programming , because that is precisely what we do when we compose deep neural nets. For example, here's how you compose a neural net consisting of two "dense" layers (linear transformations), using Keras's functional API, and then apply these two layers to some tensor x to obtain a tensor y: f = Dense(n) g = Dense(n) y = f(g(x)) This l…

NN are _just_ transfer functions. Look up tables. Or really dense maps. So f(g(x)) make total sense. But I dont think these are the interesting combinations. I think giving one NN the training experience of another, plus the feedback on "correct inference" will be when on NN trains its replacement.

Re: “Deep Learning has outlived its usefulness as a buzz-phrase”

#80

Post by sitkack » Sat, Jan 06, 2018, 7:44 AM UTC

seanmcdirmid wrote: post #57

contextfree wrote: post #54

Does this mean programming language nerds get to play too, maybe after boning up on our calculus and topology?

They already are; e.g. Jeff Dean among many others. The question is will the PL academic community play as well. Conal Elliott has done a lot of work in this area about 10 years ago. His work is beautiful but maybe before it’s time.

Can you expand on this comment, I don't understand.