Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

newsweek.com

Post reply on HN
Print view View original

91–100 of 147 posts

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#91

Post by belter »

fragmede wrote: post #84
belter wrote: post #82

Earlier quoted context omitted.

Actually yes...If the kid spends their whole life in the box and never invents a new block, that’s just combinatorics. We don’t call a chess engine ‘creative’ for finding novel moves, because we understand the rules. LLMs have rules too, they’re called weights. I want LLMs to create, but so far, every creative output I’ve seen is just a clever remix of training data. The most advanced models still fail a simple test:…

The combinatorics on choosing 500 pieces (words) out of a bag of 1.8 million pieces (approx parameters per layer for GPT-3) with replacement, and order matters works out to be something like 10^4600. Maybe you can't call that creativity, but you've got to admit that's a pretty big number.

I said No handwaving with scale. :-)

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#92

Post by balamatom »

jxjnskkzxxhx wrote: post #47
antirez wrote: post #41

Earlier quoted context omitted.

LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). So they don't learn probabilities, they learn how to continue the sentence with a given token. We apply softmax at the logits for mathematical reasons, and it is natural/simpler to think in terms of probabilities, but that's not what happens, nor the neural networks they are composed of is…

I don't think the difference is material, between "they learn probabilities" Vs "they learn how they want a sentence to continue". Seems like an implementation detail to me. In fact, you can add a temperature, set it to zero, and you become deterministic, so no probabilities anywhere. The fact is, they learn from examples of sequences and are very good at finding patterns in those sequences, to a point that they "sou…

I don't find anything surprising about that. What humans generally see of each other is little more than outer shells that are made out of sequenced linguistic patterns. They generally find that completely sufficient.

(All things considered, you may be right to be suspicious of me.)

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#93

Post by ksec »

fragmede wrote: post #75
SJC_Hacker wrote: post #59

Earlier quoted context omitted.

The list of great minds who thought that "new fangled thing is nonsense" and later turned out to be horribly wrong is quite long and distinguished

> Heavier-than-air flying machines are impossible. -Lord Kelvin. 1895 > I think there is a world market for maybe five computers. Thomas Watson, IBM. 1943 > On talking films: “They’ll never last.” -Charlie Chaplin. > This ‘telephone’ has too many shortcomings… -William Orton, Western Union. 1876 > Television won’t be able to hold any market -Darryl Zanuck, 20th Century Fox. 1946 > Louis Pasteur’s theory of germs is r…

I am just wondering did you have this all somehow saved up or did you pull it out of somewhere? Amazing list of things. Thank You.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#94

Post by timewizard »

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

> So at this point it does not matter what you believe about LLMs: in general, to trust LeCun words is not a good idea.

One does not follow from the other. In particular I don't "trust" anyone who is trying to make money off this technology. There is way more marketing than honest science happening here.

> and o3 did huge progresses on ARC,

It also cost huge money. The cost increase to go from 75% to 85% was two orders of magnitude greater. This cost scaling is not sustainable. It also only showed progress on ARC1, which it was trained for, and did terribly on ARC2 which it was not trained for.

> Btw, other researchers that were in the LeCun side, changed side recently,

Which "side" researchers are on is the least useful piece of information available.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#95

Post by timewizard »

onlyrealcuzzo wrote: post #5
ejang0 wrote: post #3

"[Yann LeCun] believes [current] LLMs will be largely obsolete within five years."

Obsolete by? This seems like a broken clock having a good chance of being right. There's so much progress, it wouldn't be that surprising if something quite different completely overtakes the current trend within 5 years .

Obsolete by price. This technology only scales linearly. All the investment in it had a different growth expectation. I suspect this level of investment will eventually collapse.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#96

Post by mdp2021 »

bko wrote: post #42
gcr wrote: post #25

Earlier quoted context omitted.

Why is changing one’s mind when confronted with new evidence a negative signifier of reputation for you?

Because he has a core belief and based on that core belief he made some statements that turned out to be incorrect. But he kept the core belief and adjusted the statements. So it's not so much about his incorrect predictions, but that these predictions were based on a core belief. And when the predictions turned out to be false, he didn't adjust his core beliefs, but just his predictions. So it's natural to ask, if n…

I have not followed all of LeCun's past statements, but -

if the "core belief" is that the LLM architecture cannot be the way to AGI, that is more of an "educated bet", which does not get falsified when LLMs improve but still suggest their initial faults. If seeing that LLMs seem constrained in the "reactive system" as opposed to a sought "deliberative system" (or others would say "intuitive" vs "procedural" etc.) was an implicit part of the original "core belief", then it still stands in spite of other improvements.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#97

Post by wat10000 »

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

LLMs literally are just predicting tokens with a probabilistic model. They’re incredibly complicated and sophisticated models, but they still are just incredibly complicated and sophisticated models for predicting tokens. It’s maybe unexpected that such a thing can do summarization, but it demonstrably can.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#98

Post by mdp2021 »

antirez wrote: post #26
gcr wrote: post #25

Earlier quoted context omitted.

Why is changing one’s mind when confronted with new evidence a negative signifier of reputation for you?

Because there were plenty of evidences that the statements were either not correct or not based on enough information, at the time they were made. And to be wrong because of personal biases, and then don't clearly state you were wrong when new evidenced appeared, is not a trait of a good scientist. For instance: the strong summarization abilities where already something that, alone, without any further information, w…

> strong summarization abilities

Which LLMs have shown you "strong summarization abilities"?

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#99

Post by gibsonf1 »

csdvrx wrote: post #88
gibsonf1 wrote: post #76

Earlier quoted context omitted.

My point is no type of math will work to model reason. Math is one of the many tools of reason, it is not the basis for reason. This is a very common error.

> My point is no type of math will work to model reason Then I disagree with you.

Exactly, we disagree and you are not alone in thinking this. You can use reason to do math, but your can't model reason with math.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#100

Post by MrMcCall »

SJC_Hacker wrote: post #59
danielmarkbruce wrote: post #57

Earlier quoted context omitted.

If you need basically rock solid evidence of X before you stop saying "this thing cannot do X", then you shouldn't be running a forward looking lab. There are only so many directions you can take, only so many resources at your disposal. Your intuition has to be really freakishly good to be running such a lab. He's done a lot of amazing work, but his stance on LLMs seems continuously off the mark.

The list of great minds who thought that "new fangled thing is nonsense" and later turned out to be horribly wrong is quite long and distinguished

I doubt that list is as long as the great minds that glommed onto a new tech that turned out to be a dud, but I could be wrong. It's an interesting question, but each tech needs to be evaluated separately.
Post reply on HN