Attention Wasn't All We Needed

Post by achierius » Sat, May 24, 2025, 5:28 PM UTC

rusuereboutdat wrote: post #19

johnsmith1840 wrote: post #17

One interesting thought process i've had around these topics is how it's not just attention but all DL methods suffer similar problems. I truly believe the last step to AGI is solving continual learning. Efficient will always inch up but the "jump" is honestly not in sight. Maybe attention + (unknown thing) really is all we need. The thought is interesting because if you extrapolate that all DL models suffer the same…

For the same reason Yann LeCun and everyone else says language won’t lead to AGI, nothing will lead to AGI. Yann says language models need to be updated with new language to describe new observation. But that’s not just with language. That’s physics. We cannot solve going to Mars or anything without the process. But space time is endless and eventually some composition of it will come along the continuous learning ma…

Post by BriggyDwiggs42 » Mon, May 26, 2025, 4:56 PM UTC

johnsmith1840 wrote: post #22

BriggyDwiggs42 wrote: post #20

Earlier quoted context omitted.

I just… why can’t it adapt over time?

Nobody knows. It's one of those seemingly simple problems to which the solutions imply contradicting answers.

Post by BriggyDwiggs42 » Wed, May 28, 2025, 2:31 AM UTC

johnsmith1840 wrote: post #24

BriggyDwiggs42 wrote: post #23

Earlier quoted context omitted.

I just don’t get why we’re talking about cosmic scales but modern AI tech and not a hypothetical ASI a thousand years out with an iq of 2 million that would actually encounter these limits.

Yeah that's not what I was talking about I was only talking about continual learning. Just hijacked thr comment because my focus is on CL on current systems not the hypothetical.