Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

81–90 of 148 posts

Re: Why deep learning works even though it shouldn’t

#81

Post by taneq »

UncleOxidant wrote: post #64
blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

Doesn't the loss function landscape depend a lot on what you're trying to get the neural network to learn (what problem you're trying to solve)?

Of course, but people seem to generally find it’s “egg-carton shaped” rather than “mountain range shaped”.

Re: Why deep learning works even though it shouldn’t

#82

Post by cwyers »

p1esk wrote: post #78
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

Good tricks or intuitions don't stay hidden for long. Everything good is quickly published, can easily be found in high quality implementations, and is discussed on GH issues, Pytorch forums, r/MachineLearning, or Twitter.

That just kicks the can down the road a step, though: that just makes the folklore knowing where to find the good GitHub issues.

Re: Why deep learning works even though it shouldn’t

#83

Post by moron4hire »

brundolf wrote: post #76
wenc wrote: post #74

Earlier quoted context omitted.

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

Other fields have their popular message boards. It's not just programmers.

Re: Why deep learning works even though it shouldn’t

#84

Post by m0zg »

brundolf wrote: post #57

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to th…

This is true in a lot of other fields. This is also the reason why experienced people make a lot more money than inexperienced ones. They know where the rakes are buried under the leaves.

Re: Why deep learning works even though it shouldn’t

#85

Post by Animats »

Now that's fascinating.

I'd thought of machine learning as a form of optimization. Things like support vector machines really were hill climbing for some kind of local optimum point. But at a billion dimensions, you're doing something else entirely. I once went through Andrew Ng's old machine learning course on video, and he was definitely doing optimization.

The last time I actually had to do numerical optimization using gradients, I was trying to solve nonlinear differential equations for a physics engine in about a 20-dimensional space of joint angles that was very "stiff". That is, some dimensions might be many orders of magnitude steeper than another. It's like walking on a narrow zigzagging mountain ridge without falling off.

So deep learning is not at all like either of those. Hm.

Re: Why deep learning works even though it shouldn’t

#86

Post by brundolf »

wenc wrote: post #77
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

I feel the word "toxic" is maybe a bit too pejorative for what it is? Another framing is that it is "guild knowledge". The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. That said, there are certain open avenues for making such knowledge public. MathOverflow is one. Some academics document their guild knowledge in "technical papers" which they put up…

> The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge.

> Sometimes this leads to public feuds however, and some academic communities are small enough that if you make too many enemies your publications may be visited upon with disfavor when it comes time for peer review. It's not worth getting into public tiffs unless there's a principle at stake.

This is more the kind of thing I was using the word "toxic" to describe. Of course I know this is a widespread and deep-seeded problem and not one that could be fixed overnight, I was just commenting on it

Re: Why deep learning works even though it shouldn’t

#87

Post by hobofan »

brundolf wrote: post #76
wenc wrote: post #74

Earlier quoted context omitted.

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

There is a venue for that stuff and it's in blog posts. The main problem however is that for a hype topic like this there are so many blog posts that it's hard to find the gems with valueable insights between all of the badly regurgitated common knowledge.

Re: Why deep learning works even though it shouldn’t

#88

Post by p1esk »

cwyers wrote: post #82
p1esk wrote: post #78

Earlier quoted context omitted.

Good tricks or intuitions don't stay hidden for long. Everything good is quickly published, can easily be found in high quality implementations, and is discussed on GH issues, Pytorch forums, r/MachineLearning, or Twitter.

That just kicks the can down the road a step, though: that just makes the folklore knowing where to find the good GitHub issues.

The point is, the experts do freely share this knowledge. Making it accessible to "casual reader" would be low on my priority list.

Re: Why deep learning works even though it shouldn’t

#89

Post by jellyksong »

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

This article is about optimization (finding good parameters), not the approximation power of neural networks (which is well-known through the universal approximation theorem).

Re: Why deep learning works even though it shouldn’t

#90

Post by dnautics »

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

> if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing.

No, this is actually exactly the opposite of what the article is saying.

"If you have enough lines you can approximate any curve within a margin" is usually a bad thing because your approximations lose meaning as the number of lines increase.

The surprising thing about machine learning is that increasing the "number of lines", if you will, increases the meaning, too, and there's some wierd and subtle properties about mathematics in higher dimensions that makes this work.

Post reply on HN