Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

141–150 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#141

Post by hansvm »

bilsbie wrote: post #127
soVeryTired wrote: post #116

Earlier quoted context omitted.

Whenever people bring this up I like to remind them that linear interpolation is a universal function approximator.

Can you expand on that?

I'll use 1NN as the interpolation strategy instead since I think it illustrates the same point and saves a few characters.

Recap: 1NN says that given a query Q you choose any pair (X,Y) from your learned "model" (a finite set of (X,Y) pairs) M minimizing |Q-X|. Your output is Y.

The following kind of argument works for linear interpolation too (you can even view 1NN as 1-point interpolation), but it's ever so slightly messier since definitions vary a fair bit, you potentially need to talk about the existence of >1 discrete "nearest" or "enclosing" set of neighbors, and proving that you can get away with fewer points than 1NN or have lower error than 1NN is itself also messier.

Pick your favorite compact-domain, continuous function embedded in some Euclidean space. For any target error you'd like to hit, the uniform continuity of that function guarantees that if your samples cover the domain well enough (no point in the domain is greater than some fixed distance, needing smaller distances for lower errors, from some point in your model) then the maximum error from a 1NN strategy is bounded by the associated error given by uniform continuity (which, again, you can make as small as you'd like by increasing the sampling resolution). The compact domain means you can physically achieve those error bounds with finite sample sizes.

For a simple example, imagine fitting more and more, smaller and smaller, line segments to y=x^2 on [-1,1].

Re: There Will Be a Scientific Theory of Deep Learning

#142

Post by vunderba »

coppsilgold wrote: post #23
pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

Comparing Deep Learning with neuroscience may turn out to be erroneous. They may be orthogonal. The brain likely has more in common with Reservoir Computing (sans the actual learning algorithm) than Deep Learning. Deep Learning relies on end to end loss optimization, something which is much more powerful than anything the brain can be doing. But the end-to-end limitation is restricting, credit assignment is a big pro…

> If only we could train a model to just use Photoshop directly, but we can't.

They're obviously more general purpose but LLMs can also be used to drive external graphics programs. A relatively popular one is Blender MCP [1], which lets an LLM control Blender to build and scaffold out 3D models.

[1] - https://github.com/ahujasid/blender-mcp

Re: There Will Be a Scientific Theory of Deep Learning

#143

Post by hansvm »

andbberger wrote: post #60
chadcmulligan wrote: post #53

Earlier quoted context omitted.

"why do neural networks work better than other models?" That sounds really interesting - any references (for a non specialist)?

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

Interestingly, there exist problems which provably can't be learned via gradient descent for them.

Re: There Will Be a Scientific Theory of Deep Learning

#144

Post by disgruntledphd2 »

hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

> the question "why do neural networks work better than other models?" is getting pretty close to a solid answer.

This would be great, as from the "classical" perspective, the results of over-parametization and potentially other parts of NN architecture make no sense (to me, at least). I do accept that double-descent appears to empirically work, but it really, really shouldn't. In fact, as someone who's a big fan of Hastie et al's Elements, the bias variance tradeoff suggests that they shouldn't.

This has been bugging me (sporadically) for years, and any progress towards an answer would be incredibly useful (most probably in a philosophical sense I suppose).

As an aside, I've only read the Introduction, but this appears to be a well-written paper and a research program I can get behind. I really want this stuff to work.

I guess it's similar to bagging and boosting, which were empirically successful well before we had any theoretical understanding of why they work.

Re: There Will Be a Scientific Theory of Deep Learning

#145

Post by krisoft »

r0ze-at-hn wrote: post #99
hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

> One can take a great 70B model and have it run in only ~16GB with no loss in capability and the ability to keep training, but the last few years funding only went for "bigger".

Awesome. What is holding you back? What do you need the funding for?

Re: There Will Be a Scientific Theory of Deep Learning

#146

Post by seydor »

It's interesting how we use one learning tool (our brain) to understand another. There is a question about what is the goal of learning mechanics: SGD already works well enough and making it a few times better will still not answer fundamental questions about what the black boxes do (rather than how they learn), because in many ways our brains are also black boxes. I think this was missing some links from learning mechanics to psychology and indeed philosophical ideas about the nature of thought and language.

Re: There Will Be a Scientific Theory of Deep Learning

#147

Post by seydor »

slashdave wrote: post #47
le-mark wrote: post #22

> We argue complexity conceals underlying regularity, and that deep learning will indeed admit a scientific theory That would be amazing, but personally I’m skeptical.

There is an analogy with statistical mechanics. It's not crazy.

biology thrives in complexity though; yet all the electrons are identical

Re: There Will Be a Scientific Theory of Deep Learning

#148

Post by seydor »

A_D_E_P_T wrote: post #37
stared wrote: post #28

Well, "There Will Be a Scientific Theory of Deep Learning" looks like flag planting - an academic variant of "I told you so!", but one that is a citation magnet.

It's actually really fascinating that there isn't a scientific theory of deep learning, especially as it's a product of human engineering as opposed to e.g. biology or particle physics.

Well there are some fundamentals such as universal approximation.

And we may find that biology also exploits structures like deep nets

Post reply on HN