Does Deep Learning Have Deep Flaws? • Hacker News Board

Re: Does Deep Learning Have Deep Flaws?

#51

Post by wantrepeneur » Fri, Jun 20, 2014, 7:54 AM UTC

agibsonccc wrote: post #21

JabavuAdams wrote: post #10

Earlier quoted context omitted.

The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.

I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE

That library looks like garbage. The website describes convnets as a "type of restricted Boltzmann machine." How can anyone trust a library with that level of misinformation?

Re: Does Deep Learning Have Deep Flaws?

#52

Post by varelse » Fri, Jun 20, 2014, 3:37 PM UTC

dekhn wrote: post #26

varelse wrote: post #14

This problem was observed 20+ years ago with linear models used for protein structure prediction. For any given model of what described a properly folded protein, one could locate conformations of the same protein that were rated as folded even better than the correct conformation (I called them doppelgangers, but the name "decoy" is what caught on). The statistical naivete of the field led to all sorts of inadverten…

Thanks for saying this. I can't really comment on your experiments (I;m not qualified) but you can be assured that some people are working in machine learning today specifically having learnt the lessons of pre- and post-CASP. I don't know that I agree CASP was founded specifically because people found decoys, but... it was an special shock when I learned about ensemble methods (I think they were just called "combine…

> don't know that I agree CASP was founded specifically because people found decoys, but...

Here's an example of what drove my work back then:

Look at the energies and RMSDs (a measure of distance from the native structure) of melittin in these two papers:

Table 2 in http://onlinelibrary.wiley.com/doi/10.1002/pro.5560020508/pd...

and

Table 1 in http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1260499/pdf/biop...

In the first paper, the energy is higher, but the RMSD is lower. In the second paper, the RMSD is higher, but the energy is lower. How did this happen?

Well, in the first paper, phi/psi angles are set directly from a library of sequentially homologous dipeptides to pentapeptides that INCLUDES MELITTIN. So, by the time you get to tripeptides, you're nearly guaranteed to just be outputting the native conformation phi/psi angles over and over again. And this paper is just one of many to make basic mistakes like this.

As young turk back then, I got into a rather long and vigorous online argument with one of the founders of CASP who insisted the first paper was a partial solution to the protein folding problem. And I suspect that argument influenced the subsequent creation of CASP.

Anyway, it's been nice rehashing my post-doc glory days(tm), but we no longer have any excuses here. We have the tools, we have the technology...

Re: Does Deep Learning Have Deep Flaws?

#53

Post by agibsonccc » Fri, Jun 20, 2014, 5:18 PM UTC

wantrepeneur wrote: post #51

agibsonccc wrote: post #21

Earlier quoted context omitted.

I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE

That library looks like garbage. The website describes convnets as a "type of restricted Boltzmann machine." How can anyone trust a library with that level of misinformation?

I appreciate the feedback. I need to clarify the implementation in DL4j in the documentation.

The impl is a convolutional RBM. That being said, where's your deep learning library? ;)

The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad.

Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think anyone who judges the quality of a lib based on a 2 second reading isn't qualified to judge much. A lot of it does rely on me with the documentation, but I'd love to chat with you one on one if you think I'm not qualified. Trash me all day, I'll either learn something or embarrass you. Both are fine with me ;).

Re: Does Deep Learning Have Deep Flaws?

#54

Post by wantrepeneur » Fri, Jun 20, 2014, 7:03 PM UTC

agibsonccc wrote: post #53

wantrepeneur wrote: post #51

Earlier quoted context omitted.

That library looks like garbage. The website describes convnets as a "type of restricted Boltzmann machine." How can anyone trust a library with that level of misinformation?

I appreciate the feedback. I need to clarify the implementation in DL4j in the documentation. The impl is a convolutional RBM. That being said, where's your deep learning library? ;) The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad. Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think…

A lot of people like Torch. I haven't looked at pylearn2 in a while, but that might be good too. Then there are a few researcher libraries with limited documentation. Nitish Srivastava has his DeepNet library and George Dahl has gdbn and there are certainly a few others too. Who knows, maybe people will start contributing documentation.

I would probably recommend Torch at this point. The incentives don't exist for the experts to make really good open source projects and spend all the time required maintaining them and helping people with them.

Re: Does Deep Learning Have Deep Flaws?

#55

Post by agibsonccc » Fri, Jun 20, 2014, 7:08 PM UTC

wantrepeneur wrote: post #54

agibsonccc wrote: post #53

Earlier quoted context omitted.

I appreciate the feedback. I need to clarify the implementation in DL4j in the documentation. The impl is a convolutional RBM. That being said, where's your deep learning library? ;) The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad. Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think…

A lot of people like Torch. I haven't looked at pylearn2 in a while, but that might be good too. Then there are a few researcher libraries with limited documentation. Nitish Srivastava has his DeepNet library and George Dahl has gdbn and there are certainly a few others too. Who knows, maybe people will start contributing documentation. I would probably recommend Torch at this point. The incentives don't exist for th…

That's fine. I do this full time. Despite it being new, I'm coming at it from a stand point of providing a platform for newer users and apps around it. You would be surprised the demand for industry.

You're right about this which is why I started a company around it.

I've already talked with andrew ng and yoshua bengio. My incentives are different from there's, however, I do have their blessings to continue doing this.

I walked in to this expecting skeptics. That being said, I love deep learning as a field and will be implementing every possible neural net I can. Since my incentives are different, I can explore the different use cases with customers and help further the field in different directions that might not make sense for say baidu, facebook ,or google.

[1] http://www.skymind.io/