Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#301

Post by inimino »

zamubafoo wrote: post #120
inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

Honest question: Given that the only wide consensus of anything approaching general intelligence are humans and that humans are biological systems that have evolved in physical reality, is there any arguments that better efficiency is even possible without relying on leveraging the nature of reality? For example, analog computers can differentiate near instantly by leveraging the nature of electromagnetism and you ca…

Inference is already wasteful (compared to humans) but training is absurd. There's strong reason to believe we can do better (even prior to having figured out how).

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#302

Post by inimino »

fluidcruft wrote: post #133
jedberg wrote: post #76

Earlier quoted context omitted.

> and biologically implausible I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains. Is this a common thing in ML papers or something you came up with?

How are you separating the efficiency of the architecture from the efficiency of the substrate? Unless you have a brain made of transistors or an LLM made of neurons how can you identify the source of the inefficiency?

You can't but the transistor-based approach is the inefficient one, and transistors are pretty good at efficiently doing logic, so either there's no possible efficient solution based on deterministic computation, or there's tremendous headroom.

I believe human and machine learning unify into a pretty straightforward model and this shows that what we're doing that ML doesn't can be copied across, and I don't think the substrate is that significant.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#303

Post by nickpsecurity »

adamnemecek wrote: post #152

We are actually working on scaling energy-based models http://traceoid.ai

I hope you succeed. I've downloaded and at least skimmed hundreds of papers ML, many alternative architectures. A subset of them built prototypes that claimed good results on benchmarks. Of those, many didn't pass further scrutiny due to various failures. Those that did pass often failed on real-world tasks despite doing well on benchmarks. That we're so jaded by failures of published models makes us even more skeptical of unpublished methods.

So, each architectural advance needs published prototypes solving real-world problems. The smallest I've seen do useful stuff are in 100+M-3B range. There are also papers about testing advances with low, pretraining cost: BabyLM; GPT2 replications; MosaicBERT. Some do straight pre-training while others distill field-proven models. Alternative architectures would do well to crank out examples like this to prove themselves.

Please, do build at least one of the above using your method. Post it to your site. Link to demos of the actual prototype in use. This might get an ecosystem going that builds on your ideas.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#304

Post by numba888 »

jcims wrote: post #203
bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer. Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slippin…

Great, but how do you imagine multimodal with text, video. Just 2 for simplicity, what will be in the training set. With text model tries to predict next, then more steps were added. But what to do with multimodal?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#305

Post by adamnemecek »

nickpsecurity wrote: post #303
adamnemecek wrote: post #152

We are actually working on scaling energy-based models http://traceoid.ai

I hope you succeed. I've downloaded and at least skimmed hundreds of papers ML, many alternative architectures. A subset of them built prototypes that claimed good results on benchmarks. Of those, many didn't pass further scrutiny due to various failures. Those that did pass often failed on real-world tasks despite doing well on benchmarks. That we're so jaded by failures of published models makes us even more skepti…

How many of these were actually addressing scaling EBMs though? I'm guessing none.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#306

Post by physicsguy »

ninetyninenine wrote: post #214
bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

>1) You can't learn an accurate world model just from text. >2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. LLMs can be trained with multimodal data. Language is only tokens and pixel and sound data can be encoded into tokens. All data can be serialized. You can train this thing on data we can't even comprehend. Here's the big question. It's clear we…

Hugely different data too?

If you think of all the neurons connected up to vision, touch, hearing, heat receptors, balance, etc. there’s a constant stream of multimodal data of different types along with constant reinforcement learning - e.g. ‘if you move your eye in this way, the scene you see changes’, ‘if you tilt your body this way your balance changes’, etc. and this runs from even before you are born, throughout your life.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#307

Post by numeri »

littlestymaar wrote: post #55
Lerc wrote: post #50

Earlier quoted context omitted.

I feel like we're stacking naive misinterpretations of how LLMs function on top of one another here. Grasping gradient descent and autoregressive generation can give you a false sense of confidence. It is like knowing how transistors make up logic gates and believing you know more than CPU design than you actually do. Rather than inferring from how you imagine the architecture working, you can look at examples and co…

No an ML researcher or anything (I'm basically only a few Karpathy video into ML, so please someone correct me if I'm misunderstanding this), but it seems that you're getting this backwards: > One misconception is that predicting the next word means there is no internal idea on the word after next. The simple disproof of this is that models put 'an' instead of 'a' ahead of words beginning with vowels. My understandin…

No, the person you're responding to is absolutely right. The easy test (which has been done in papers again and again) is the ability to train linear probes (or non-linear classifier heads) on the current hidden representations to predict the nth-next token, and the fact that these probes have very high accuracy.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#308

Post by YeGoblynQueenne »

kadushka wrote: post #211
iainctduncan wrote: post #208

Earlier quoted context omitted.

Thanks for articulating this so well. I'm a musician and music/CS phd student, and as a jazz improvisor of advanced skill (30+ years), I'm accutely aware that there are significant areas of intelligence for which linguistic thinking is not only not good enough, but something to be avoided as much as one can (which is bloody hard sometimes). I have found it so frustrating, but hard to figure out how to counter, that t…

Most modern LLMs are multimodal.

Tri-modal at best: text, sound and video, and that's it. That's just barely "multi" (it's one more than two).

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#309

Post by YeGoblynQueenne »

throw310822 wrote: post #159
bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

I don't get it. 1) Yes it's true, learning from text is very hard. But LLMs are multimodal now. 2) That "size of a lion" paper is from 2019, which is a geological era from now. The SOTA was GPT2 which was barely able to spit out coherent text. 3) Have you tried asking a mouse to play chess or reason its way through some physics problem or to write some code? I'm really curious in which benchmark are mice surpassing c…

Oh mice can solve a plethora of physics problems before it's time for breakfast. They have to navigate the, well, physical world, after all.

I'm also really curious what benchmarks LLMs have passed that include surviving without being eaten by a cat, or a gull, or an owl, while looking for food to survive and feed one's young in an arbitrary environment chosen from urban, rural, natural etc, at random. What's ChatGPT's score on that kind of benchmark?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#310

Post by randomNumber7 »

eximius wrote: post #295
randomNumber7 wrote: post #229

Earlier quoted context omitted.

Why, even animals sleep? And if you for example learn an instrument you will notice that a lot of the learning of the muscel memory happens during sleep.

I guess you're saying that non-inference time training can be that "sleep period"?

Yes, i could imagine something like a humanoid robot, where the "short term memory" is just a big enough context to keep all input of the day. Then during "sleep" there is training where the information is processed.

But I also think that current LLM tech does not lead to agi. You cant train something on pattern matchin and then it becomes magically intelligent (although i could be wrong).

Imo an AGI would need to be able to interact with the environment and learn to reflect on its interactions and its abilities within it. I suspect we have the hardware to build s.th. intelligent as a cat or a dog, but not the algorithms.

Post reply on HN