Does anyone know if there is any code available for this paper? Would be extremely interesting to see comparisons of the EM-algorithm mentioned with regular SGD in terms of speed and precision.
A Probabilistic Theory of Deep Learning
11–20 of 21 posts
Re: A Probabilistic Theory of Deep Learning
#1256 pages! They should really reorganize this into 10-16 pages to get the basic ideas and results across.
What a strange attitude! There's plenty of zero-cost bits available to allow for articles of all sorts of lengths. For example, the "Foundations and Trends" journals (e.g. Foundations and Trends in Machine Learning, or Foundations and Trends in Optimization) publish articles that are usually at least 100 pages long. These articles tend to be well-respected and highly cited.
Longer articles have many advantages in allowing for a more in-depth explanation, and it is certainly not the case that every reader wants papers shoe-horned into an artificial page limit.
Re: A Probabilistic Theory of Deep Learning
#13What is the hypothesis? How was the hypothesis tested?
From my reading of the first twenty or so pages, it appears to be a theory of how neural networks model images (the running example). The authors claim that popular neural network architectures can be reduced to particular cases of their model. If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocki…
> If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocking
This paper [0] does a pretty good job of giving an explanation for that.
Re: A Probabilistic Theory of Deep Learning
#14What is the hypothesis? How was the hypothesis tested?
From my reading of the first twenty or so pages, it appears to be a theory of how neural networks model images (the running example). The authors claim that popular neural network architectures can be reduced to particular cases of their model. If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocki…
The adversarial noise issue is not so hard to understand, it's just costly to correct.
That link provides the explanation: if your classifier is not very regularized, then the classification regions are going to be close and irregular, s.t. a small vector may lead you from one to another. It's more of a geometrical fact f you think of classification regions in those spaces (of high dimension).
Guaranteeing a large minimum distance is hard (essentially why error correcting codes are pretty hard to encode/decode)
Re: A Probabilistic Theory of Deep Learning
#1556 pages! They should really reorganize this into 10-16 pages to get the basic ideas and results across.
Arbitrary page limits cause far more harm than good in academic writing.
Dissertations and journal articles are some of the most readable and useful academic publications in computer science precisely because this attitude of "16 pages of bits ought to be enough for anyone" isn't enforced.
Without artificial page limits, it's possible to explain an idea from the ground up without taking shortcuts. Enforcing a prohibitive page limit is far more likely to force clever writing than good communication.
Re: A Probabilistic Theory of Deep Learning
#16Earlier quoted context omitted.
From my reading of the first twenty or so pages, it appears to be a theory of how neural networks model images (the running example). The authors claim that popular neural network architectures can be reduced to particular cases of their model. If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocki…
> If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocking This paper [0] does a pretty good job of giving an explanation for that. [0] http://arxiv.org/abs/1412.6572
Honestly, that abstract makes me more upset. If these are due to NN's nature as linear classifers, then we are all in trouble, given that almost everything useful is based off linear models. Given the title of the paper though, I should probably be more hopeful :)
Re: A Probabilistic Theory of Deep Learning
#1756 pages! They should really reorganize this into 10-16 pages to get the basic ideas and results across.
What a strange attitude! There's plenty of zero-cost bits available to allow for articles of all sorts of lengths. For example, the "Foundations and Trends" journals (e.g. Foundations and Trends in Machine Learning, or Foundations and Trends in Optimization) publish articles that are usually at least 100 pages long. These articles tend to be well-respected and highly cited. Longer articles have many advantages in all…
> There's plenty of zero-cost bits available to allow for articles of all sorts of lengths.
Straw man - i wasn't talking about cost, hard drive space, or any relation near what you're referring to.
> ...articles that are usually at least 100 pages long. These articles tend to be well-respected and highly cited.
Another straw man - why does length (short or long) correlate with quality again?
> Longer articles have many advantages in allowing for a more in-depth explanation
Ah the real comment. Ok. I definitely agree - longer usually means more space to explain.
> and it is certainly not the case that every reader wants papers shoe-horned into an artificial page limit.
So be it. There's usually an appendix or supplementary materials that can offer expanded derivations. Often the authors trim a lot of the fat for the published paper and put a longer version in a book/thesis.
> What a strange attitude!
As a writer of publications i want more space and agree, but as a reader of publications (way more than i write) there's just too much out there to spend my time going through 50+ pages. I can put in the time for 10-20 pages and if i still want more i'll check out other publications, appendix, supplementary, thesis...whatever. It's an important and necessary skill for academics to be able to concisely present their work - not just for publications, but for grant applications, presentations, etc.
Re: A Probabilistic Theory of Deep Learning
#18Earlier quoted context omitted.
What a strange attitude! There's plenty of zero-cost bits available to allow for articles of all sorts of lengths. For example, the "Foundations and Trends" journals (e.g. Foundations and Trends in Machine Learning, or Foundations and Trends in Optimization) publish articles that are usually at least 100 pages long. These articles tend to be well-respected and highly cited. Longer articles have many advantages in all…
> There's plenty of zero-cost bits available to allow for articles of all sorts of lengths. Straw man - i wasn't talking about cost, hard drive space, or any relation near what you're referring to. > ...articles that are usually at least 100 pages long. These articles tend to be well-respected and highly cited. Another straw man - why does length (short or long) correlate with quality again? > Longer articles have ma…
If you don't like to read long articles, don't read them. I like to read them; they tend to be much easier to understand than artificially shortened articles. It seems to me that complaining that somebody wrote a long article because they take too long for you to read is exactly analogous to saying that since you only like to read short stories, nobody should write a long novel because it would take too much time for you to read.
Re: A Probabilistic Theory of Deep Learning
#1956 pages! They should really reorganize this into 10-16 pages to get the basic ideas and results across.
Maybe you should stick to twitter.
Re: A Probabilistic Theory of Deep Learning
#20What is the hypothesis? How was the hypothesis tested?
From my reading of the first twenty or so pages, it appears to be a theory of how neural networks model images (the running example). The authors claim that popular neural network architectures can be reduced to particular cases of their model. If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocki…
the fact that random noise can be classified as strongly belonging to some class, and the fact that classification results can be unstable, is simply a result of the fact the input space is very high dimensional, and the output space is very small (say a few isolated points). That is, if you are discriminatively training a mapping from images in R^(224 x 224 x 3) to 1000 points (class labels), there is going to be a tremendous amount of instability in the inverse direction.