Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

111–120 of 148 posts

Re: Why deep learning works even though it shouldn’t

#111

Post by renewiltord »

brundolf wrote: post #76
wenc wrote: post #74

Earlier quoted context omitted.

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

All of this stuff already exists. By design it excludes people who would likely disparage the lack of rigor. And sometimes other people get caught in that net.

That's a good way. It's called knowing your audience. No one wants to be savaged on HN or Reddit by a bunch of people.

Re: Why deep learning works even though it shouldn’t

#112

Post by machinelearning »

bigyikes wrote: post #108
machinelearning wrote: post #97

Earlier quoted context omitted.

While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum. These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur. From a researcher's standpoi…

> I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This seems super meta, because I am a beginner engineer and your comment is totally pattern matching to distill…

If you want to go more meta, I can point out that your comment is pattern matching to training models and applying them to humans vs. thinking from first principles and realizing that humans and models learn differently. That's exactly the issue with applying functional truths as truths.

Re: Why deep learning works even though it shouldn’t

#113

Post by brandmeyer »

Fordec wrote: post #98
mrfusion wrote: post #94

This quote: there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to the casual reader. I’d love to hear these intuitions from every field. Anyone got some?

All sensor data, the closer you get to the analog side of things, is bullshit. It's just about smoothing over the bullshit enough to make the tolerances workable for real world applications. We call this bullshit smoothing "calibration". If you're doing work on sensor data and don't have every calibration parameter, whether from configuration or magic factory numbers and statistical tolerances, someone, somewhere is…

> Ever looked at weather data from two separate apps and have values vary by multiple degrees?

Actually this is because surface temperatures really do have this kind of variation.

Re: Why deep learning works even though it shouldn’t

#114

Post by woopwoop »

moultano wrote: post #106
woopwoop wrote: post #52

It seems to me all of these arguments apply just as well when the "deep" network has only one hidden layer.

That's true. I left out saying that deeper networks represent a wider variety of functions more easily, because that seems generally intuitive to everyone. But the arguments about how easy it is to optimize them should apply equally well to wide and shallow networks as to very deep ones.

Thanks for the response. Hmm, it's still pretty mysterious to me. Why should a deep network with the same number of parameters as a wide network represent a wider variety of functions? In some sense they represent the "same" number of functions, in the sense that the manifold of functions given by two network architectures with the same number of parameters have have the same dimension, even if one is wide while the other is deep.

Re: Why deep learning works even though it shouldn’t

#115

Post by philipov »

moron4hire wrote: post #83
brundolf wrote: post #76

Earlier quoted context omitted.

It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?

Other fields have their popular message boards. It's not just programmers.

What are the popular message boards for practicing physicists and mathematicians?

Re: Why deep learning works even though it shouldn’t

#116

Post by KKKKkkkk1 »

But it doesn't. Researchers have been saying for several years now that computer vision is more accurate than human vision, and face recognition was one of the first problems "solved." And yet when the pandemic hit, Apple had to scramble to adjust its unlock mechanism in iOS 13.5 because Face ID cannot recognize people wearing masks [1]. Humans have no trouble identifying people wearing masks. We are now almost a year into the pandemic, iOS 14 has been released, Face ID has not been fixed, and now we hear that Apple is bringing Touch ID back [2].

So sure, you've developed a methodology that can overfit nicely not only the train data but even the test data. But it still fails miserably when you apply your model in the field.

[1] https://www.theverge.com/2020/5/20/21265019/apple-ios-13-5-o...

[2] https://appleinsider.com/articles/20/10/16/under-display-tou...

Re: Why deep learning works even though it shouldn’t

#118

Post by dehrmann »

wenc wrote: post #74
brundolf wrote: post #57

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to th…

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impressi…

> Reputations are at stake, and most people are doing impression management

What's the saying about academia? The politics are so vicious because stakes are so small? It's pretty ironic because the stakes are larger for a lot of decisions made by engineers in industry, but we seem content with swapping best practices.

Re: Why deep learning works even though it shouldn’t

#119

Post by why_only_15 »

KKKKkkkk1 wrote: post #116

But it doesn't. Researchers have been saying for several years now that computer vision is more accurate than human vision, and face recognition was one of the first problems "solved." And yet when the pandemic hit, Apple had to scramble to adjust its unlock mechanism in iOS 13.5 because Face ID cannot recognize people wearing masks [1]. Humans have no trouble identifying people wearing masks. We are now almost a yea…

I don't think this is quite fair. Of course a machine learning model doesn't work on things it wasn't trained for. Computer vision is not more accurate than human vision in every possible field. And are you really sure humans have no trouble identifying people wearing masks? Lots of people look pretty similar, especially if all you can see is their face.

Re: Why deep learning works even though it shouldn’t

#120

Post by rajansaini »

KKKKkkkk1 wrote: post #116

But it doesn't. Researchers have been saying for several years now that computer vision is more accurate than human vision, and face recognition was one of the first problems "solved." And yet when the pandemic hit, Apple had to scramble to adjust its unlock mechanism in iOS 13.5 because Face ID cannot recognize people wearing masks [1]. Humans have no trouble identifying people wearing masks. We are now almost a yea…

Intuitively, it seems like voice signatures, body language (like walking habits), and height/weight would play a larger role in helping humans identify a masked person than exposed facial features.

Because the network behind Apple's facial recognition software cannot have access to this kind of data (well, maybe voice, but that doesn't seem secure), I'm not sure this is a fair comparison.

Would love to be refuted, however.

Post reply on HN