Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generalization in Deep Learning [pdf]

arxiv.org

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Generalization in Deep Learning [pdf]

#11

Post by seanmcdirmid »

sgt101 wrote: post #5
gfredtech wrote: post #2

An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?

read the abstract, read the conclusion, try to work out what the authors are trying to say.. hunt for the pay off in the paper, see if I can understand that, look for the results and check to see if I can understand that what they claim to show supports the pay off then start reading it. The actual reading can take up to six weeks for a big journal paper. True.

Skim, index, find context (e.g. Citations to the work if old enough). Some papers take a few years to really understand, which is why mental indexing is important: when you run into a problem solved by the paper, the paper becomes useful and understandable at the same time.

Re: Generalization in Deep Learning [pdf]

#12

Post by Eridrus »

sgt101 wrote: post #6

The Generalization Gap (test/train) seems to me to be quite unsatisfying as a definition, and then an appeal that theory and practice are different doesn't make me feel that anything is explained. Being able to create new regularisation techniques is impressive though... looks like something that will need a read!

I haven't read the paper yet, but when people say things like "We don't know why neural networks work", what they mean is we don't understand why they generalise so we'll, by most prior theory they are quite over parameterized, so while they should be able to fit the data - and it has been shown that typical architectures can fit random data - previous theory said the penalty for this is that it will not generalize well to new data from the same distribution.

So generalization bounds on NNs are actually the key thing that people want from theory.

Re: Generalization in Deep Learning [pdf]

#13

Post by steev »

gfredtech wrote: post #2

An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?

Proof-heavy papers can take days or weeks to understand. It takes multiple reads, working through the math on scrap paper, and frequently discussing proofs with colleagues.

My process is generally to read through the paper a couple of times and then start tackling the proofs (if there are any). In general, I find it best to start with a very high level understanding and slowly descend into a greater detailed understanding.

Re: Generalization in Deep Learning [pdf]

#14

Post by sgt101 »

seanmcdirmid wrote: post #11
sgt101 wrote: post #5

Earlier quoted context omitted.

read the abstract, read the conclusion, try to work out what the authors are trying to say.. hunt for the pay off in the paper, see if I can understand that, look for the results and check to see if I can understand that what they claim to show supports the pay off then start reading it. The actual reading can take up to six weeks for a big journal paper. True.

Skim, index, find context (e.g. Citations to the work if old enough). Some papers take a few years to really understand, which is why mental indexing is important: when you run into a problem solved by the paper, the paper becomes useful and understandable at the same time.

Agree about the years to understand - the important thing is to read it enough so that when you come across something you remember the approach and can option your investment in it.

Re: Generalization in Deep Learning [pdf]

#17

Post by benrbray »

amelius wrote: post #10
gfredtech wrote: post #2

An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?

> how long does it take you to read one paper? It depends on to what extent the paper is self-contained.

By applying Lemma 4.9 of Lee 2014 to Corollary 6.2.11 of Simpson 1954, but substituting Tao 1977's simplified integral expression in the denominator, the result is obvious.
Post reply on HN