Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

91–100 of 148 posts

Re: Why deep learning works even though it shouldn’t

#91

Post by sjg007 »

brundolf wrote: post #76
wenc wrote: post #74

Earlier quoted context omitted.

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

Require people to publish their code.

Re: Why deep learning works even though it shouldn’t

#92

Post by tbenst »

blackbear_ wrote: post #20
RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression.

https://arxiv.org/abs/1806.06850

Re: Why deep learning works even though it shouldn’t

#93

Post by moultano »

hobofan wrote: post #87
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

There is a venue for that stuff and it's in blog posts. The main problem however is that for a hype topic like this there are so many blog posts that it's hard to find the gems with valueable insights between all of the badly regurgitated common knowledge.

I'm curious which category you put this one in. :D

Re: Why deep learning works even though it shouldn’t

#94

Post by mrfusion »

This quote:

there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to the casual reader.

I’d love to hear these intuitions from every field. Anyone got some?

Re: Why deep learning works even though it shouldn’t

#95

Post by Fishysoup »

bArray wrote: post #79
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

Personally I've wanted a 1-page IEEE publication for a while that accepts smaller contributions, where people can share these kinds of insights. Just a "we tried this, this is what happened" or "we were not able to repeat this" or "we found this interesting, but we need more data". You could argue it's kind of like a long abstract, but a long abstract really indicates you intend to probe it further, but in actual fac…

People have been floating the idea of doing this for the life sciences too, where the cost of performing an experiment can be very high (animals, enzymes, local permitting and legal considerations etc. on top of the man hours). It would be very useful to see the results of an experiment you want to do(or something resembling it) but didn't follow up on it for whatever reason and would definitely accelerate research. And would probably help with the weird incentive structure and secrecy issues academia is pointlessly built around.

Re: Why deep learning works even though it shouldn’t

#96

Post by moultano »

tbenst wrote: post #92
blackbear_ wrote: post #20

Earlier quoted context omitted.

You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850

Let us know when polynomial regression succeeds at any machine learning task.

A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.

Re: Why deep learning works even though it shouldn’t

#97

Post by machinelearning »

brundolf wrote: post #57

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to th…

While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum.

These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur.

From a researcher's standpoint, learning these is counterproductive if the goal is to study and understand the underlying mechanisms from first principles.

From a beginner engineer perspective, these intuitions may be effective "functional truths" but there's the danger of perceiving these handwavy intuitions as truths. This leads to inflexibility in light of empirical evidence that contradicts these intuitions and even worse - not debugging enough since the pattern seems to match roughly the intuition. The latter results in flawed institutional knowledge being accrued over time. An engineer might say: "Engineer X tried Y and it didn't work because of ML intuition Z, since this is a related problem, we should not prioritize Y due to the precedent."

I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This will help them avoid making the mistakes that people who rely on this intuitive language too much, mistaking it for ML theory.

Re: Why deep learning works even though it shouldn’t

#98

Post by Fordec »

mrfusion wrote: post #94

This quote: there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to the casual reader. I’d love to hear these intuitions from every field. Anyone got some?

All sensor data, the closer you get to the analog side of things, is bullshit. It's just about smoothing over the bullshit enough to make the tolerances workable for real world applications.

We call this bullshit smoothing "calibration". If you're doing work on sensor data and don't have every calibration parameter, whether from configuration or magic factory numbers and statistical tolerances, someone, somewhere is pulling the wool over the eyes of the software guy downstream that works with the final data.

Ever looked at weather data from two separate apps and have values vary by multiple degrees? Two different pipeline just sprinkled on their own versions of interpretations on top of raw sewage data.

Re: Why deep learning works even though it shouldn’t

#99

Post by kerng »

wenc wrote: post #77
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

I feel the word "toxic" is maybe a bit too pejorative for what it is? Another framing is that it is "guild knowledge". The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. That said, there are certain open avenues for making such knowledge public. MathOverflow is one. Some academics document their guild knowledge in "technical papers" which they put up…

That's why publishing and presenting outside the formal academic realm is important. Especially presentations that have one level of indirection from original creators often provide much better intuition and also the presenter isn't afraid sharing that he/she doesn't know certain things or they present content in a more creative funny way.

Re: Why deep learning works even though it shouldn’t

#100

Post by ivalm »

blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

If anything for large models I thought the idea is everything is a saddle point. Your link looks at a relatively small dense network.
Post reply on HN