There Will Be a Scientific Theory of Deep Learning • Hacker News Board

Re: There Will Be a Scientific Theory of Deep Learning

#151

Post by robotresearcher » Sat, Apr 25, 2026, 4:05 PM UTC

mystraline wrote: post #43

pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

Ive yet to see a model that trains AND applies the trained data real-time. Thats basically every living being, from bacteria to plants to mammals. Even PID loops have a training phase separate from recitation phase.

Reinforcement learning on real robots in real time has been done lots of times, since back in the 90s at least. It’s painfully slow.

Re: There Will Be a Scientific Theory of Deep Learning

#152

Post by fulafel » Sat, Apr 25, 2026, 4:12 PM UTC

Interesting conceptual parallel to vibecoding, where also you get something that works, and it's another job to understand how or why.

Re: There Will Be a Scientific Theory of Deep Learning

#153

Post by shevy-java » Sat, Apr 25, 2026, 4:53 PM UTC

> In this paper, we make the case that a scientific theory of deep learning is emerging

Wait ... they created something they do not understand and can not yet describe?

And this is now called ... science?

Remember that they "borrowed" words from biology, in particular neurobiology, decades ago already. Monkey see, monkey do copy paste.

Re: There Will Be a Scientific Theory of Deep Learning

#154

Post by trinari » Sat, Apr 25, 2026, 5:05 PM UTC

pdhborges wrote: post #102

r0ze-at-hn wrote: post #99

Earlier quoted context omitted.

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

Can you share that document?

also interested

Re: There Will Be a Scientific Theory of Deep Learning

#155

Post by mystraline » Sat, Apr 25, 2026, 5:34 PM UTC

robotresearcher wrote: post #151

mystraline wrote: post #43

Earlier quoted context omitted.

Ive yet to see a model that trains AND applies the trained data real-time. Thats basically every living being, from bacteria to plants to mammals. Even PID loops have a training phase separate from recitation phase.

Reinforcement learning on real robots in real time has been done lots of times, since back in the 90s at least. It’s painfully slow.

Why is it slow?

We know a human uses roughly 100 watts. And teaching a new specific task takes only showing maybe 10 times to get to 80%.

The learning function in humans are definitely connected with both training/recitation.

I'm seeing that as the big roadblock between thinking machines and a really big autocomplete we have now.

Re: There Will Be a Scientific Theory of Deep Learning

#156

Post by tim333 » Sat, Apr 25, 2026, 7:39 PM UTC

j_bum wrote: post #56

RyanShook wrote: post #5

Here's where I'm missing understanding: for decades the idea of neural networks had existed with minimal attention. Then in 2017 Attention Is All You Need gets released and since then there is an exponential explosion in deep learning. I understand that deep learning is accelerated by GPUs but the concept of a transformer could have been used on much slower hardware much earlier.

This video gives a great overview of the history of the acceleration: https://youtu.be/glWvwvhZkQ8?si=-HGtfd_KHYfatEQ Although it’s focused on Ilya, some great history is covered.

That was interesting.

Re: There Will Be a Scientific Theory of Deep Learning

#157

Post by ChrisClark » Sat, Apr 25, 2026, 8:08 PM UTC

hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

You have to remember, this website is also full of managers and 'CEOs' and such that think they know more than any expert in the field.

Re: There Will Be a Scientific Theory of Deep Learning

#158

Post by Terr_ » Sat, Apr 25, 2026, 9:05 PM UTC

gopher_space wrote: post #130

r0ze-at-hn wrote: post #99

Earlier quoted context omitted.

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

> It’s honestly a bit painful watching the AI field struggle to re-learn first principles that other disciplines have already learned. This is my fear with software development in general. There's a hundred-year old point of view right next door that'll solve problems and I'm too incurious to see it. I have a relative with a focus in math education that I've been stealing ideas from, and I think we'd both appreciate…

I think some of it has to do with incentives. Nobody wants to invest in a team to adapt and test other-field lessons that may come out as "there's no free lunch" or "this is equivalent to a hard problem they didn't solve there yet either."

So instead we're more likely to see navel-gazing "singularity" stories that fit with telling your investors they will become fantastically rich.

Re: There Will Be a Scientific Theory of Deep Learning

#159

Post by gavinray » Sat, Apr 25, 2026, 9:13 PM UTC

pdhborges wrote: post #102

r0ze-at-hn wrote: post #99

Earlier quoted context omitted.

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

Can you share that document?

Crackpot "Universal Theory of Everything" physics rooted in numerology:

https://meta-r0ze.github.io/Informational-Energetics/Informa...

Re: There Will Be a Scientific Theory of Deep Learning

#160

Post by saulpw » Sat, Apr 25, 2026, 9:20 PM UTC

krisoft wrote: post #145

r0ze-at-hn wrote: post #99

Earlier quoted context omitted.

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

> One can take a great 70B model and have it run in only ~16GB with no loss in capability and the ability to keep training, but the last few years funding only went for "bigger". Awesome. What is holding you back? What do you need the funding for?

Presumably $100m to train the 70B model? I think you're assuming that the author meant you can take an existing 70B model and run it in 16GB. But it stands to reason that "no loss in capability" means it had to be trained under those constraints.