Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete • Hacker News Board

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#41

Post by antirez » Sat, Apr 05, 2025, 4:52 PM UTC

jxjnskkzxxhx wrote: post #34

antirez wrote: post #26

Earlier quoted context omitted.

Because there were plenty of evidences that the statements were either not correct or not based on enough information, at the time they were made. And to be wrong because of personal biases, and then don't clearly state you were wrong when new evidenced appeared, is not a trait of a good scientist. For instance: the strong summarization abilities where already something that, alone, without any further information, w…

I don't see the contradiction between "stochastic parrot" and "strong summarisation abilities". Where I'm skeptical of LLM skepticism is that people use the term "stochastic parrot" disparagingly, as if they're not impressed. LLMs are stochastic parrots in the sense that they probabilistically guess sequences of things, but isn't it interesting how far that takes you already? I'd never have guessed. Fundamentally I q…

LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). So they don't learn probabilities, they learn how to continue the sentence with a given token. We apply softmax at the logits for mathematical reasons, and it is natural/simpler to think in terms of probabilities, but that's not what happens, nor the neural networks they are composed of is just able to approximate probabilistic functions. This "next token" probability is the source of a lot misunderstanding. It's much better to imagine the logits as "To continue my reply I could say this word, more than the others, or maybe that one, a bit less, ..." and so forth. Now there are evidences, too, that in the activations producing a given token the LLM already has an idea about how most of the sentence is going to continue.

Of course, as they learn, early in the training, the first functions they will model, to lower the error, will start being the probabilities of the next tokens, since this is the simplest function that works for the loss reduction. Then gradients agree in other directions, and the function that the LLM eventually learn is no longer related to probabilities, but to the meaning of the sentence and what it makes sense to say next.

It's not be chance that often the logits have a huge signal in just two or three tokens, even if the sentence, probabilistically speaking, could continue in much more potential ways.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#42

Post by bko » Sat, Apr 05, 2025, 4:53 PM UTC

gcr wrote: post #25

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

Why is changing one’s mind when confronted with new evidence a negative signifier of reputation for you?

Because he has a core belief and based on that core belief he made some statements that turned out to be incorrect. But he kept the core belief and adjusted the statements.

So it's not so much about his incorrect predictions, but that these predictions were based on a core belief. And when the predictions turned out to be false, he didn't adjust his core beliefs, but just his predictions.

So it's natural to ask, if none of the predictions you derived from your core belief come true, maybe your core belief isn't true.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#43

Post by darkwater » Sat, Apr 05, 2025, 4:56 PM UTC

jaggederest wrote: post #30

antirez wrote: post #26

Earlier quoted context omitted.

Because there were plenty of evidences that the statements were either not correct or not based on enough information, at the time they were made. And to be wrong because of personal biases, and then don't clearly state you were wrong when new evidenced appeared, is not a trait of a good scientist. For instance: the strong summarization abilities where already something that, alone, without any further information, w…

Here's a fun example of that kind of "I've updated my statements but not assessed any of my underlying lack of understanding" - it's a bad look on any kind of scientist. https://x.com/AukeHoekstra/status/1507047932226375688

Who are you referring to?

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#44

Post by aprilthird2021 » Sat, Apr 05, 2025, 4:57 PM UTC

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

> As LLMs do things thought to be impossible before

Like what?

Your timeline doesn't sound crazy outlandish. It sounds pretty normal and lines up with my thoughts as AI has advanced over the past few years. Maybe more conservative than others in the field, but that's not a reason to dismiss him entirely any more than the hypesters should be dismissed entirely because they were over promising and under delivering?

> Now reasoning models can solve problems they never saw

This is not the same as a novel question though.

> o3 did huge progresses on ARC

Is this a benchmark? O3 might be great, but I think the average person's experience with LLMs matches what he's saying, it seems like there is a peak and we're hitting it. It also matches what Ilya said about training data being mostly gone and new architectures (not improvements to existing ones) needing to be the way forward.

> LeCun is directing an AI lab that as the same point has the following huge issues

Second point has nothing to do with the lab and more to do with Meta. Your last point has nothing to do with the lab at all. Meta also said they will have an agent that codes like a junior engineer by the end of the year and they are clearly going to miss that prediction, so does that extra hype put them back in your good books?

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#45

Post by djoldman » Sat, Apr 05, 2025, 4:58 PM UTC

The idolatry and drama surrounding LeCun, Hinton, Schmidhuber, etc. is likely a distraction. This includes their various predictions.

More interesting is their research work. JEPA is what LeCun is betting on:

https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-jo...

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#46

Post by aprilthird2021 » Sat, Apr 05, 2025, 5:00 PM UTC

GMoromisato wrote: post #18

I remember reading Douglas Hofstadter's Fluid Concepts and Creative Analogies [ https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_An... ] He wrote about Copycat, a program for understanding analogies ("abc is to 123 as cba is to ???"). The program worked at the symbolic level, in the sense that it hard-coded a network of relationships between words and characters. I wonder how close he was to "inventing" an L…

> In fact, pattern-matching is all there is: That's a bear, run away; I'm in a restaurant, I need to order; this is like a binary tree, I can solve it recursively.

This is not correct. It does not explain creativity at all. It cannot solely be based on pattern matching. I'm not saying no AI is creative, but this logic does not explain creativity

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#47

Post by jxjnskkzxxhx » Sat, Apr 05, 2025, 5:02 PM UTC

antirez wrote: post #41

jxjnskkzxxhx wrote: post #34

Earlier quoted context omitted.

I don't see the contradiction between "stochastic parrot" and "strong summarisation abilities". Where I'm skeptical of LLM skepticism is that people use the term "stochastic parrot" disparagingly, as if they're not impressed. LLMs are stochastic parrots in the sense that they probabilistically guess sequences of things, but isn't it interesting how far that takes you already? I'd never have guessed. Fundamentally I q…

LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). So they don't learn probabilities, they learn how to continue the sentence with a given token. We apply softmax at the logits for mathematical reasons, and it is natural/simpler to think in terms of probabilities, but that's not what happens, nor the neural networks they are composed of is…

I don't think the difference is material, between "they learn probabilities" Vs "they learn how they want a sentence to continue". Seems like an implementation detail to me. In fact, you can add a temperature, set it to zero, and you become deterministic, so no probabilities anywhere. The fact is, they learn from examples of sequences and are very good at finding patterns in those sequences, to a point that they "sound human".

But the point of my response was just that I find it an extremely surprising how well an idea as simple as "find patterns in sequences" actually works for the purpose of sounding human, and I'm suspicious of anyone who pretends this isn't incredible. Can we agree on this?

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#48

Post by aprilthird2021 » Sat, Apr 05, 2025, 5:02 PM UTC

falcor84 wrote: post #36

asdev wrote: post #21

outside of text generation and search, LLMs have not delivered any significant value

I personally have greatly benefitted from LLM's helping me reason about problems and make progress on many diverse issues across professional, recreational and mental health difficulties. I think that asking whether it's just "text generation and search" rather than something that transcends it is as meaningful as asking whether an airplane really "flies" or just "applies thrust and generates lift".

I personally benefit from AI auto complete when coding. I think that value isn't worth what people say it is. But I'm often wrong about what I think the value of something is (e.g. social media) and it's true value. I unfortunately often have people try to justify their thinking or beliefs by just pasting a ChatGPT output or screenshot to "prove" it.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#49

Post by DrBenCarson » Sat, Apr 05, 2025, 5:03 PM UTC

sorcerer-mar wrote: post #31

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

> there is no probabilistic link between the words of a text and the gist of the content How could that possibly be true? There’s obviously a link between “[original content] is summarized as [summarized”content]

It’s not true

The idea that meaning is not impacted by language yet is somehow exclusively captured by language is just absolutely absurd

Like saying X+Y=Z but changing X or Y won’t affect Z

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#50

Post by andoando » Sat, Apr 05, 2025, 5:05 PM UTC

guhidalg wrote: post #32

GMoromisato wrote: post #18

I remember reading Douglas Hofstadter's Fluid Concepts and Creative Analogies [ https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_An... ] He wrote about Copycat, a program for understanding analogies ("abc is to 123 as cba is to ???"). The program worked at the symbolic level, in the sense that it hard-coded a network of relationships between words and characters. I wonder how close he was to "inventing" an L…

I wouldn't call pattern matching intelligence, I would call it something closer to "trainability" or "educatable" but not intelligence. You can train a person to do a task without understanding why they have to do it like that, but when confronted with a new never-before-seen situation they have to understand the physical laws of the universe to find a solution. Ask ChatGPT to answer something that no one on the inte…

What precludes pattern matching from understanding the physical laws? You see a ball hit a wall, and it bounces back. Congratulations, you learned the abstract pattern:

x->|

x|

x<-|