Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

71–80 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#71

Post by ButlerianJihad »

alasdair_ wrote: post #70
cdavid wrote: post #57

Earlier quoted context omitted.

Indeed. I would add a third factor to compute and datasets: the lego-like aspect of NN that enabled scalable OSS DL frameworks. I did some ML in mid 2000s, and it was a PITA to reuse other people code (when available at all). You had some well known libraries for SVM, for HMM you had to use HTK that had a weird license, and otherwise looking at experiments required you to reimplement stuff yourself. Late 2000s had a…

I think your post may have more acronyms than any other post I have ever read on hn. Do you have a guide to which specific things you are talking about with each acronym? Deep Learning and Machine Learning are obvious but some of the others I can’t follow at all - they could be so many different things.

I think he maintains pinball machines and jukeboxes for a chain of Greek restaurants

Re: There Will Be a Scientific Theory of Deep Learning

#72

Post by hodgehog11 »

fc417fc802 wrote: post #69
andbberger wrote: post #60

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

I don't follow. Why wouldn't it work? It seems to me that a biased random walk down a gradient is about as universal as it gets. A bit like asking why walking uphill eventually results in you arriving at the top.

It wouldn't work if your landscape has more local minima than atoms in the known universe (which it does) and only some of them are good. Neural networks can easily fail, but there's a lot of things one can do to help ensure it works.

Re: There Will Be a Scientific Theory of Deep Learning

#73

Post by hodgehog11 »

jmalicki wrote: post #66
andbberger wrote: post #60

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

The properties that the uniform approximation theorem proves are not unique to neural networks. Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course). So the universal approximation theorem tells us nothing about why should expect ne…

Extremely well said. Universal approximation is necessary but not sufficient for the performance we are seeing. The secret sauce is implicit regularization, which comes about analogously to enforcing compression.

Re: There Will Be a Scientific Theory of Deep Learning

#74

Post by appplication »

hodgehog11 wrote: post #72
fc417fc802 wrote: post #69

Earlier quoted context omitted.

I don't follow. Why wouldn't it work? It seems to me that a biased random walk down a gradient is about as universal as it gets. A bit like asking why walking uphill eventually results in you arriving at the top.

It wouldn't work if your landscape has more local minima than atoms in the known universe (which it does) and only some of them are good. Neural networks can easily fail, but there's a lot of things one can do to help ensure it works.

Not a mathematician so I’m immediately out of my depth here (and butchering terminology), but it seems, intuitively, like the presence of a massive amount of local minima wouldn’t really be relevant for gradient descent. A given local minimum would need to have a “well” at least be as large as your step size to reasonably capture your descent.

E.g. you could land perfectly on a local minima but you won’t stay the unless your step size was minute or the minima was quite substantial.

Re: There Will Be a Scientific Theory of Deep Learning

#75

Post by bubblyworld »

skydhash wrote: post #68
hodgehog11 wrote: post #65

Earlier quoted context omitted.

No it isn't, and it's frustrating when the "common wisdom" tries to boil it down to this. If this was true, then the models with "infinitely many" parameters would be amazing. What about just training a gigantic two-layer network? There is a huge amount of work trying to engineer training procedures that work well. The actual reason is due to complex biases that arise from the interaction of network architectures and…

> The actual reason is due to complex biases that arise from the interaction of network architectures and the optimizers and persist in the regime where data scales proportionally to model size. The multiscale nature of the data induces neural scaling laws that enable better performance than any other class of models can hope to achieve. That’s a lot of words to say that, if you encode a class of things as numbers, t…

That isn't what they are saying at all, lol.

Re: There Will Be a Scientific Theory of Deep Learning

#77

Post by wwarner »

slashdave wrote: post #47
le-mark wrote: post #22

> We argue complexity conceals underlying regularity, and that deep learning will indeed admit a scientific theory That would be amazing, but personally I’m skeptical.

There is an analogy with statistical mechanics. It's not crazy.

Sane & interesting enough to have been disproven, by Boaz Barak iirc. Maybe not surprising since simulated annealing never achieved the results of gradient descent + backprop.

Re: There Will Be a Scientific Theory of Deep Learning

#79

Post by AkiraHsieh »

Theory becomes critical when you need to predict failure modes. A decision support system that 'just works' most of the time but fails silently on edge cases is worse than a simpler system with known limitations. Understanding the bias mechanisms would help us know when a model is confident vs when it's just pattern matching. That distinction matters when the stakes are high.

Re: There Will Be a Scientific Theory of Deep Learning

#80

Post by anvuong »

hodgehog11 wrote: post #72
fc417fc802 wrote: post #69

Earlier quoted context omitted.

I don't follow. Why wouldn't it work? It seems to me that a biased random walk down a gradient is about as universal as it gets. A bit like asking why walking uphill eventually results in you arriving at the top.

It wouldn't work if your landscape has more local minima than atoms in the known universe (which it does) and only some of them are good. Neural networks can easily fail, but there's a lot of things one can do to help ensure it works.

A funny thing is, in very high-dimensional space, like millions and billions of parameters, the chance that you'd get stuck in a local minima is extremely small. Think about it like this, to be stuck in a local minima in 2D, you only need 2 gradient components to be zero, in higher dimension, you'd need every single one of them, millions up millions of them, to be all zero. You'd only need 1 single gradient component to be non-zero and SGD can get you out of it. Now, SGD is a stochastic walk on that manifold, not entirely random, but rather noisy, the chance that you somehow walk into a local minima is very very low, unless that is a "really good" local minima, in a sense that it dominates all other local minimas in its neighborhood.
Post reply on HN