Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

101–110 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#101

Post by hodgehog11 »

geokon wrote: post #100
mathisfun123 wrote: post #59

Earlier quoted context omitted.

> why do neural networks work better than other models The only people for whom this is an open question are the academics - everyone else understands it's entirely because of the bagillions of parameters.

normally more parameters leads to overfitting (like fitting a polynomial to points), but neural nets are for some reason not as susceptible to that and can scale well with more parameters. Thats been my understanding of the crux of mystery. Would love to be corrected by someone more knowledgable though

This absolutely was the crux of the (first) mystery, and I would argue that "deep learning theory" really only took off once it recognized this. There are other mysteries too, like the feasibility of transfer learning, neural scaling laws, and now more recently, in-context learning.

Re: There Will Be a Scientific Theory of Deep Learning

#102

Post by pdhborges »

r0ze-at-hn wrote: post #99
hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

Can you share that document?

Re: There Will Be a Scientific Theory of Deep Learning

#103

Post by sanxiyn »

calf wrote: post #52

Is there not some Rice's Theorem equivalent for deep nets? After all they are machines that are randomly generated, so from classical computer science I would not presume a theory of "what do all deep nets do" to be prima facie logically possible. Nor do I see this explained in the objections section.

As I understand, Rice's theorem does not apply because neural networks are not Turing-complete.

Re: There Will Be a Scientific Theory of Deep Learning

#104

Post by energy123 »

hodgehog11 wrote: post #98
niksmather wrote: post #84

Earlier quoted context omitted.

Do neural networks work better than other models? They can definitely model a wider class of problems than traditional ML models (images being the canonical example). However, I thought where a like for like comparison was possible they tend to worse than gradient boosting.

Gradient boosting handles tabular data better than neural networks, often because the structure is simpler, and it becomes more of an issue to deal with the noise. You can do like-to-like comparisons between them for unstructured data like images, audio, video, text, and a well-designed NN will mop the floor with gradient boosting. This is because to handle that sort of data, you need to encode some form of bias arou…

Would you agree/disagree with the following:

- It's not gradient boosting per se that's good on tabular data, it's trees. Other fitting methods with trees as the model are also usually superior to NNs on tabular data.

- Trees are better on tabular data because they encode a useful inductive bias that NNs currently do not. Just like CNNs or ViTs are better on images because they encode spatial locality as an inductive bias.

Re: There Will Be a Scientific Theory of Deep Learning

#105

Post by energy123 »

porcoda wrote: post #20
RyanShook wrote: post #5

Here's where I'm missing understanding: for decades the idea of neural networks had existed with minimal attention. Then in 2017 Attention Is All You Need gets released and since then there is an exponential explosion in deep learning. I understand that deep learning is accelerated by GPUs but the concept of a transformer could have been used on much slower hardware much earlier.

As others pointed out, the explosion of interest started with the deep convolutional networks that were applied in image problems. What I always thought was interesting was that prior to that, NNs were largely dismissed as interesting. When I took a course on them around the year 2000 that was the attitude most people took. It seems like what it took to spark renewed interest was ImageNet and seeing what you get when…

Deepmind solving Atari games was another big milestone around that time.

Re: There Will Be a Scientific Theory of Deep Learning

#106

Post by imtringued »

ngruhn wrote: post #96
NooneAtAll3 wrote: post #85

Earlier quoted context omitted.

Universal approximation is like saying that a problem is computable sure, that gives some relief - but it says nothing in practice unlike f.e. which side of P/NP divide the problem is on

> unlike f.e. which side of P/NP divide the problem is on Actually the P/NP divide is a similar case in my opinion. In practice a quadratic algorithm is sometimes unacceptably slow and an NP problem can be virtually solved. E.g. SAT problems are routinely solved at scale.

An NP problem can contain subproblems that are not worst case problems.

It's similar to the gap between pushdown automata and Turing machines. You can check if pushdown automata will terminate or not. You can't do it for Turing machines, but this doesn't stop you from running a pushdown automata algorithm on the turning machine with decidable termination.

Re: There Will Be a Scientific Theory of Deep Learning

#107

Post by imtringued »

charcircuit wrote: post #88
anvuong wrote: post #80

Earlier quoted context omitted.

A funny thing is, in very high-dimensional space, like millions and billions of parameters, the chance that you'd get stuck in a local minima is extremely small. Think about it like this, to be stuck in a local minima in 2D, you only need 2 gradient components to be zero, in higher dimension, you'd need every single one of them, millions up millions of them, to be all zero. You'd only need 1 single gradient component…

>you'd need every single one of them, millions up millions of them, to be all zero If they were all correlated with each other that does not seem far fetched.

Ok but it's already known that you shouldn't initialize your network parameters to a single constant and instead initialize the parameters with random numbers.

Re: There Will Be a Scientific Theory of Deep Learning

#108

Post by jimmypk »

hodgehog11 wrote: post #73
jmalicki wrote: post #66

Earlier quoted context omitted.

The properties that the uniform approximation theorem proves are not unique to neural networks. Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course). So the universal approximation theorem tells us nothing about why should expect ne…

Extremely well said. Universal approximation is necessary but not sufficient for the performance we are seeing. The secret sauce is implicit regularization, which comes about analogously to enforcing compression.

[flagged]

Re: There Will Be a Scientific Theory of Deep Learning

#110

Post by imtringued »

jvanderbot wrote: post #27
coppsilgold wrote: post #23

Earlier quoted context omitted.

Comparing Deep Learning with neuroscience may turn out to be erroneous. They may be orthogonal. The brain likely has more in common with Reservoir Computing (sans the actual learning algorithm) than Deep Learning. Deep Learning relies on end to end loss optimization, something which is much more powerful than anything the brain can be doing. But the end-to-end limitation is restricting, credit assignment is a big pro…

If you think a 2 year old is doing deep learning, you're probably wrong. But if you think natural selection was providing end to end loss optimization, you might be closer to right. An _awful lot_ of our brain structure and connectivity is born, vs learned, and that goes for Mice and Men.

I don't know what you mean by end to end loss optimization in particular, but if you mean something that involves global propagation of errors e.g. backpropagation you are dead wrong.

Predictive coding is more biologically plausible because it uses local information from neighbouring neurons only.

Post reply on HN