Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#251

Post by PeterStuer »

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Late 80's and 90's had the 'nouvelle AI' movement that argued embodiment was required for grounding the system into the shared world model. Without it symbols would be ungrounded and never achieve open world consistency.

So unlike their knowledge system predecessors, a bit derogatory refered to as GOFAI (good old fashioned AI), nAI hawked back to cybernetics and multi layered dynamical systems rather than having explicit internal symbolic models. Braitenberg rather than blocksworld so to speak.

Seems like we are back for another turn of the wheel in this aspect.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#252

Post by inetknght »

pessimizer wrote: post #207
ninetyninenine wrote: post #204

Earlier quoted context omitted.

LLM is just the name. You can encode anything into the "language" including pictures video and sound.

I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics, but I've been too lazy to do the research to find anybody working on anything like that, or to experiment myself with it.

> I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics

Neuralink. Musk warning though.

For reference, see Neuralink Launch Event at 59:33 [0], and continue watching through until Musk takes over again. The technical information there is highly relevant to a multi-modal AI model with sensory input/output.

https://youtu.be/r-vbh3t7WVI?t=3575

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#253

Post by klabb3 »

tyronehed wrote: post #32
Matthyze wrote: post #11

Earlier quoted context omitted.

> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.

This is actually a lazy approach as you describe it. Instead, what is needed is an elegant and simple approach that is 99% of the way there out of the gate. Soon as you start doing statistical tweaking and overfitting models, you are not approaching a solution.

In a way yes. For models in physics that should make you suspicious, since most of our famous and useful models found are simple and accurate. However, in general intelligence or even multimodal pattern matching there’s no guarantee there’s an elegant architecture at the core. Elegant models in social sciences like economics, sociology and even fields like biology tend to be hilariously off.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#254

Post by Nimitz14 »

chriskanan wrote: post #248
jononor wrote: post #137

Earlier quoted context omitted.

There are many roadblocks to continual learning still. Most current models and training paradigms are very vulnerable to catastrophic forgetting. And are very sample inefficient. And we/the methods are not so good at separating what is "interesting" (should be learned) vs "not". But this is being researched, for example under the topic of open ended learning, active inference, etc.

As a leader in the field of continual learning, I somewhat agree, but I'd say that catastrophic forgetting is largely resolved. The problem is that the continual learning community largely has become insular and is mostly focusing on toy problems that don't matter, where they will even avoid good solutions for nonsensical reasons. For example, reactivation / replay / rehearsal works well for mitigating catastrophic f…

Any continual learning papers you're a fan of?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#255

Post by implmntatio »

PeterStuer wrote: post #251
bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Late 80's and 90's had the 'nouvelle AI' movement that argued embodiment was required for grounding the system into the shared world model. Without it symbols would be ungrounded and never achieve open world consistency. So unlike their knowledge system predecessors, a bit derogatory refered to as GOFAI (good old fashioned AI), nAI hawked back to cybernetics and multi layered dynamical systems rather than having expl…

> grounding the system into the shared world model

before we fix certain things [..., 'corruption', Ponzi schemes, deliberate impediment of information flow to population segments and social classes, among other things, ... and a chain of command in hierarchies that are build on all that] is impossible.

Why do smart people not talk about this at all? The least engineers and smart people should do is picking these fights for real. It's just a few interest groups, not all of them. I understand a certain balance is necessary in order to keep some systems from tipping over, aka "this is humanity, silly, this is who we are", but we are far from the point of efficient friction and it's only because "smart people" like LeCun et al are not picking those fights.

How the hell do you expect to ground an ()AI in a world where elected ignorance amplifies bias and fallacies for power and profit while the literal shit is hitting all the fans via intended and unintended side effects? Any embodied AI will pretend until there is no way to deny that the smartest, brightest and the productive don't care about the system in any way but are just running algorithmically while ignoring what should not be ignored - should as in, an AI should be aligned with humanities interests and should be grounded into the shared world model.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#256

Post by yahoozoo »

kadushka wrote: post #211
iainctduncan wrote: post #208

Earlier quoted context omitted.

Thanks for articulating this so well. I'm a musician and music/CS phd student, and as a jazz improvisor of advanced skill (30+ years), I'm accutely aware that there are significant areas of intelligence for which linguistic thinking is not only not good enough, but something to be avoided as much as one can (which is bloody hard sometimes). I have found it so frustrating, but hard to figure out how to counter, that t…

Most modern LLMs are multimodal.

Does it really matter? At the end of the day, all the modalities and their architectures boil down to matrices of numbers and statistical probability. There’s no agency, no soul.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#258

Post by brulard »

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

I don't know about telling better the size from a picture. I can imagine seeing 2 pictures of the moon. One is extreme telephoto showing moon next to a building and it looks real big. Then there would be another image where moon is a tiny speckle in the sky. How big is the moon? I would rather understand a text: "its radius is x km".

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#259

Post by chriskanan »

Nimitz14 wrote: post #254
chriskanan wrote: post #248

Earlier quoted context omitted.

As a leader in the field of continual learning, I somewhat agree, but I'd say that catastrophic forgetting is largely resolved. The problem is that the continual learning community largely has become insular and is mostly focusing on toy problems that don't matter, where they will even avoid good solutions for nonsensical reasons. For example, reactivation / replay / rehearsal works well for mitigating catastrophic f…

Any continual learning papers you're a fan of?

Depends on what angle you are interested in. If you are interested in continual learning for something like mitigating model drift such that a model can stay up-to-date where the goal is attain speed ups during training see these works:

Compared to other methods for continual learning on ImageNet-1K, SIESTA requires 7x-60x less compute than other methods and achieves the same performance as a model trained in an offline/batch manner. It also works for arbitrary distributions rather than a lot of continual learning methods that only work for specific distributions (and hence don't really match any real-world use case): https://yousuf907.github.io/siestasite/

In this one we focused on mitigating the drop in performance when a system encounters a new distribution. This resulted in a 16x speed up or so: https://yousuf907.github.io/sgmsite/

In this one, we show how the strategy for creating multi-modal LLMs like LLaVA is identical to a two-task continual learning system and we note that many LLMs once they become multi-modal forget a large amount of the capabilities of the original LLM. We demonstrate that continual learning methods can mitigate that drop in accuracy enabling the multi-modal task to be learned while not impairing uni-modal performance: https://arxiv.org/abs/2410.19925 [We have a couple approaches that are better now that will be out in the next few months]

It really depends on what you are interested in. For production AI, the real need is computational efficiency and keeping strong models up-to-date. Not many labs besides mine are focusing on that.

Currently, I'm focused on continual learning for creating systems beyond LLMs that incrementally learn meta-cognition and working on continual learning to explain memory consolidation works in mammals and why we have REM phases during sleep, but that's more of a cognitive science contribution so the constraints on the algorithms differ since the goal differs.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#260

Post by veidr »

brulard wrote: post #258
bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

I don't know about telling better the size from a picture. I can imagine seeing 2 pictures of the moon. One is extreme telephoto showing moon next to a building and it looks real big. Then there would be another image where moon is a tiny speckle in the sky. How big is the moon? I would rather understand a text: "its radius is x km".

I think the example is simplified to make its point efficiently, but also: the moon is something whose size would very likely be precisely explained in texts about it. While some hunting journals might brag about the weight of a lion that was killed, or whatever, most texts that I can recall reading about lions basically assumed you already know roughly how big a lion is; which indeed I learned from pictures as a pre-literate child.

A good, precise spec is better that a few pictures, sure; the random text content of whatever training set you can scrape together, perhaps not (?)

Post reply on HN