Ask HN: Any insider takes on Yann LeCun's push against current architectures? • Hacker News Board

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#201

Post by danielmarkbruce » Sat, Mar 15, 2025, 3:31 AM UTC

heyjamesknight wrote: post #195

danielmarkbruce wrote: post #143

Earlier quoted context omitted.

> Doesn't Language itself encode multimodal experiences Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we don't have great text datasets to describe many things in enough detail, but there isn't any reason we couldn't.

There are absolutely reasons that we cannot capture the entirety—or even a proper image—of human cognition in semantic space. Cognition is not purely semantic. It is dynamic, embodied, socially distributed, culturally extended, and conscious. LLMs are great semantic heuristic machines. But they don't even have access to those other components.

The LLM embeddings for a token cover much more than semantics. There is a reason a single token embedding dimension is so large.

You are conflating the embedding layer in an LLM and an embedding model for semantic search.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#202

Post by wavemode » Sat, Mar 15, 2025, 3:37 AM UTC

throw310822 wrote: post #45

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

> there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Have you ever tried telling ChatGPT that you're "in the city centre" and asking it if you need to turn left or right to reach some landmark? It will not answer with the average of the directions given to everybody who asked the question before, it will answer asking you…

That's because, based on the training data, the most likely response to asking for directions is to clarify exactly where you are and what you see.

But if you ask it in terms of a knowledge test ("I'm at the corner of 1st and 2nd, what public park am I standing next to?") a model lacking web search capabilities will confidently hallucinate (unless it's a well-known park).

In fact, my person opinion is that, therein lies the most realistic way to reduce hallucination rates: rather than trying to train models to say "I don't know" (which is not really a trainable thing - models are fundamentally unaware of the limits of their own training data), instead just train them on which kinds of questions warrant a web search and which ones should be answered creatively.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#203

Post by jcims » Sat, Mar 15, 2025, 3:38 AM UTC

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer.

Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slipping on a cold sidewalk can be explained in a clinical fashion, but an android’s association of that to the powerful dynamic response required to even attempt to recover will give a newfound association and power to the word ‘slip’.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#204

Post by ninetyninenine » Sat, Mar 15, 2025, 3:44 AM UTC

jcims wrote: post #203

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer. Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slippin…

LLM is just the name. You can encode anything into the "language" including pictures video and sound.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#205

Post by jongjong » Sat, Mar 15, 2025, 3:56 AM UTC

It's weird that we don't have human-level AGI yet considering that we have AIs that are in some ways much smarter than humans.

Top-end LLMs write better and faster than most humans.

Top-end stable diffusion models can draw and render video much faster and with much more precision than the best human artists.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#206

Post by prats226 » Sat, Mar 15, 2025, 3:59 AM UTC

I feel like success of LLM's have been combination of multiple factors coming together favourably: 1) Hardware becoming cheap enough to train models beyond a size where we could see emergent properties. Which is going to become cheaper and cheaper. 2) Model architecture which can in computationally less expensive manner being able to look at all inputs at the same time. CNN's, RNN's all succeded at smaller scale becuase they added inductive bias in architecture favourable to the input modality, but also became less generic. Attention is simpler in computation to scale it and also has lower inductive bias. 3) Unsupervised text on internet being source of data which requires light pre-processing hence almost no efforts wrt annotations etc reaching scale wrt scaling laws corrosponding to large size models. Also text data being diverse enough to be generic to encompass variety of topics, thoughts vs imagenet etc which is highly specific and costly to produce.

Assuming that text only models will hit a bottleneck, then to have next generation models, in addition to a new architecture, we also have to find rich dataset which is even more generic and much richer in modalities and the architecture being able to natively ingest it?

However something that is not predictible is how well the emergent properties can scale with model size further. Maybe few more unlocks like model being able to retain information well in spite of really large context length, ability to SFT on super complex reasoning tasks without disrupting weights enough to loose unsupervised learning might take us much further?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#207

Post by pessimizer » Sat, Mar 15, 2025, 4:02 AM UTC

ninetyninenine wrote: post #204

jcims wrote: post #203

Earlier quoted context omitted.

Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer. Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slippin…

LLM is just the name. You can encode anything into the "language" including pictures video and sound.

I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics, but I've been too lazy to do the research to find anybody working on anything like that, or to experiment myself with it.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#208

Post by iainctduncan » Sat, Mar 15, 2025, 4:02 AM UTC

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

Thanks for articulating this so well. I'm a musician and music/CS phd student, and as a jazz improvisor of advanced skill (30+ years), I'm accutely aware that there are significant areas of intelligence for which linguistic thinking is not only not good enough, but something to be avoided as much as one can (which is bloody hard sometimes). I have found it so frustrating, but hard to figure out how to counter, that the current LLM zeitgeist seems to hinge on a belief that linguistic intelligence is both necessary and sufficient for AGI.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#209

Post by kadushka » Sat, Mar 15, 2025, 4:15 AM UTC

chmod775 wrote: post #197

refulgentis wrote: post #172

Earlier quoted context omitted.

I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM" Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN. What in the world doe…

> What in the world does it mean that a 3 year old is smarter than an LLM? Because LLMs have terrible comprehension of the real world. Here's an example: > You: If you put a toddler next to a wig on the floor, which reaches higher? > ChatGPT: The wig would typically reach higher than the toddler, especially if the wig is a standard size or has long hair. Toddlers are generally around 2 to 3 feet tall, while wigs can…

I just entered your prompt into ChatGPT (4.5):

Prompt:

I'm trying to make a rope, made of only aubergines, not using any twine or string. What kind of knot would you recommend to tie together each aubergine with the next? I'm trying to use up as little of the length of each aubergine as possible.

Response:

Your request doesn't entirely make sense. Aubergines (eggplants) are firm but brittle vegetables; they lack flexibility, elasticity, or tensile strength required to create or sustain knots. Attempting to tie aubergines together directly, without using twine, string, or another flexible medium, isn't feasible. They'd likely break or tear apart under stress.

If your goal is genuinely to form a rope-like structure using only aubergines, you'd be much better off slicing the aubergines into thin, flexible strips (perhaps using a vegetable peeler or mandoline), drying or partially dehydrating them to increase flexibility, and then braiding or twisting them together, rather than knotting.

If you clarify your intended purpose or constraints further, I can suggest alternative methods.

The response looks good to me.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#210

Post by pessimizer » Sat, Mar 15, 2025, 4:17 AM UTC

mystified5016 wrote: post #154

codenlearn wrote: post #138

Earlier quoted context omitted.

Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentia…

Imagine I give you a text of any arbitrary length in an unknown language with no images. With no context other than the text, what could you learn? If I told you the text contained a detailed theory of FTL travel, could you ever construct the engine? Could you even prove it contained what I told you? Can you imagine that given enough time, you'd recognize patterns in the text? Some sequences of glyphs usually follow…

I'm optimistic about this. I think enough pictures of an apple, chemical analyses of the air, the ability to arbitrarily move around in space, a bunch of pressure sensors, or a bunch of senses we don't even have, will solve this. I suspect there might be a continuum of more concept understanding that comes with more senses. We're bathed in senses all the time, to the point where we have many systems just to block out senses temporarily, and to constantly throw away information (but different information at different times.)

It's not a theory of consciousness, it's a theory of quality. I don't think that something can be considered conscious that is constantly encoding and decoding things into and out of binary.