Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

101–110 of 148 posts

Re: Why deep learning works even though it shouldn’t

#101

Post by jaimex2 »

yters wrote: post #47
morei wrote: post #42

Earlier quoted context omitted.

I think voice recognition would be a better example. It went from 'toy' to 'everyday use' because the word error rate dropped an order of magnitude.

Is that due to ML or all the massive crowdsourcing and more keyword/search driven approach? e.g. has dictation apps like dragon improved substantially?

Both funnily enough. ML has been around for years, the main reason it's getting better is that it's easier than ever to collect massive amounts of data to train good models.

Re: Why deep learning works even though it shouldn’t

#102

Post by RandyRanderson »

jellyksong wrote: post #89
RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

This article is about optimization (finding good parameters), not the approximation power of neural networks (which is well-known through the universal approximation theorem).

I know. What we're both saying is: if you have enough lines, you can find the params easily.

Re: Why deep learning works even though it shouldn’t

#103

Post by RandyRanderson »

dougabug wrote: post #16
RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

ReLU networks are piecewise linear.

Have a look at some published nn architectures - you'll typically see a few nonlinear units/transforms and a LOT of ReLU ones.

Re: Why deep learning works even though it shouldn’t

#104

Post by RandyRanderson »

ngrilly wrote: post #29
RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

> if you have enough lines you can approximate any curve within a margin How avoid overfitting the training data?

You're right - a LOT of work goes into this (eg GANs) because these large nNets just start "remembering" the training data.

Re: Why deep learning works even though it shouldn’t

#106

Post by moultano »

woopwoop wrote: post #52

It seems to me all of these arguments apply just as well when the "deep" network has only one hidden layer.

That's true. I left out saying that deeper networks represent a wider variety of functions more easily, because that seems generally intuitive to everyone. But the arguments about how easy it is to optimize them should apply equally well to wide and shallow networks as to very deep ones.

Re: Why deep learning works even though it shouldn’t

#107

Post by skybrian »

salawat wrote: post #13
quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

The concept of optima is heavily dependent upon constraints on dimensionality. If you're surrounded by walls at a 3D coordinate (walls in this sense is usually something akin to an a priori constraint on step size, which itself is the upper and lower bounds of the imposed delta introduced to the current step to try to find a new direction to go in numerical gradient descent), but you can arbitrarily "jump" 4 dimensio…

Autocomplete in GPT-3 isn't optimization at all, though? The optimization was in the training.

I've read that the loops you're talking about come from taking too many high-probability choices compared to real-world text that has some low-probability words.

Re: Why deep learning works even though it shouldn’t

#108

Post by bigyikes »

machinelearning wrote: post #97
brundolf wrote: post #57

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to th…

While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum. These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur. From a researcher's standpoi…

> I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to.

This seems super meta, because I am a beginner engineer and your comment is totally pattern matching to distillation, but for humans instead of models. Maybe beginners should start with those “functional“ (read: distilled) truths and train from there :)

Re: Why deep learning works even though it shouldn’t

#109

Post by munchbunny »

brundolf wrote: post #76
wenc wrote: post #74

Earlier quoted context omitted.

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

It’s the stuff you 60-80% believe but you don’t want to put your name on it because once you do people assume you believe it 99% or more. That’s unfortunately how publishing it works.
Post reply on HN