Generalization in Deep Learning [pdf]
1–10 of 17 posts
Re: Generalization in Deep Learning [pdf]
#2An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?
Re: Generalization in Deep Learning [pdf]
#3An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?
I use a customized version of this: http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/p...
The most important thing for me is to not read it from the beginning to the end (which is hard for me). Abstract->Conclusions, scan headlines. Methods is the most curious section for me. Depending on the paper and what I am working on I read the methods section last or first (even before the abstract). If it's more of a "oh that seems neat" paper I skip the methods section and mostly extract the idea. The book "How to read a book" is also a good source of ideas.
I don't think there's a one size fits all approach. I also find the various papers on writing literature reviews very helpful (for gathering an overview of a topic). Just checked my Zotero and these are the ones I have tagged:
"Using grounded theory as a method for rigorously reviewing literature"
"On being ‘systematic’in literature reviews in IS"
"A hermeneutic approach for conducting literature reviews and literature searches"
"Systematic literature reviews in software engineering–a systematic literature review"
"Writing narrative literature reviews."
Feel free to go as meta as you want ;)
Re: Generalization in Deep Learning [pdf]
#4Why are they using MNIST to demonstrate generalization? Isn't it basically a toy problem at this point?
Re: Generalization in Deep Learning [pdf]
#5An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?
read the abstract, read the conclusion, try to work out what the authors are trying to say.. hunt for the pay off in the paper, see if I can understand that, look for the results and check to see if I can understand that what they claim to show supports the pay off then start reading it. The actual reading can take up to six weeks for a big journal paper.
True.
Re: Generalization in Deep Learning [pdf]
#6The Generalization Gap (test/train) seems to me to be quite unsatisfying as a definition, and then an appeal that theory and practice are different doesn't make me feel that anything is explained.
Being able to create new regularisation techniques is impressive though... looks like something that will need a read!
Re: Generalization in Deep Learning [pdf]
#7The Generalization Gap (test/train) seems to me to be quite unsatisfying as a definition, and then an appeal that theory and practice are different doesn't make me feel that anything is explained. Being able to create new regularisation techniques is impressive though... looks like something that will need a read!
I think the reasoning is that optimizing the training performance is "easy", whereas optimizing the test performance is "hard". If you can guarantee that test performance will be close to training performance, then optimizing the test performance becomes "easy".
Re: Generalization in Deep Learning [pdf]
#8Why are they using MNIST to demonstrate generalization? Isn't it basically a toy problem at this point?
It is a problem that is well-known and easy to solve. Their point is to show that they offer a boost in terms of training speed.
Re: Generalization in Deep Learning [pdf]
#9An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?
> how long does it take you to read one paper?
It really depends. A deep learning paper just presenting a new architecture can be read in a few hours, meanwhile for some heavy papers on the mathematical side can take days, weeks to fully digest.
In the end it doesn't matter, you can be as slow as you want :)
Re: Generalization in Deep Learning [pdf]
#10An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?
> how long does it take you to read one paper?
It depends on to what extent the paper is self-contained.