Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

newsweek.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#81

Post by mdp2021 »

baumy wrote: post #27
asdev wrote: post #21

outside of text generation and search, LLMs have not delivered any significant value

Text generation and search are the drivers for some trillions of dollars worth of economic activity around the world.

> trillions of dollars

That is monetary value. The poster may have meant "delivery" value - which has been limited (and tainted with hype).

> Text generation

Which «text generation», apart from code generation (quite successful in some models), would amount to «trillions of dollars worth of economic activity» at the current stage? I cannot see it at the moment.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#82

Post by belter »

fragmede wrote: post #79
belter wrote: post #72

Earlier quoted context omitted.

But have we established that LLMs dont just interpolate and they can create? Are we able to prove it with output that's 1) algorithmically novel (not just a recombination) 2) coherent, and 3) not explainable by training data coverage. No handwaving with scale...

Why is that the bar though? Imagine LLMs as a kid that has a box of lego with a hundred million blocks in it, and it can assemble those blocks into any configuration possible. Is the fact that the kid doesn't have access to ABS plastic pellets and a molding machine, and so they can't make new pieces; does that really make us think that the kid just interpolates and can't create?

Actually yes...If the kid spends their whole life in the box and never invents a new block, that’s just combinatorics. We don’t call a chess engine ‘creative’ for finding novel moves, because we understand the rules. LLMs have rules too, they’re called weights.

I want LLMs to create, but so far, every creative output I’ve seen is just a clever remix of training data. The most advanced models still fail a simple test: Restrict the domain, for example, "invent a cookie recipe with no flour, sugar, or eggs" or "name a company without using real words". Suddenly, their creativity collapses into either, nonsense (violating constraints), or trivial recombination, ChocoNutBake instead of NutellaCookie.

If LLMs could actually create, we’d see emergent novelty, outputs that couldn’t exist in the training data. Instead, we get constrained interpolation.

Happy to be proven wrong. Would like to see examples where an LLM output is impossible to map back to its training data.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#83

Post by neom »

DrBenCarson wrote: post #49
sorcerer-mar wrote: post #31

Earlier quoted context omitted.

> there is no probabilistic link between the words of a text and the gist of the content How could that possibly be true? There’s obviously a link between “[original content] is summarized as [summarized”content]

It’s not true The idea that meaning is not impacted by language yet is somehow exclusively captured by language is just absolutely absurd Like saying X+Y=Z but changing X or Y won’t affect Z

Language is a symbolic system. From an absolute or spiritual standpoint, meaning transcends pure linguistic probabilities. Language itself emerges as a limited medium for the expression of consciousness and abstract thought. Indeed, to say meaning arises purely from language (as probability alone) or, to deny language influences meaning entirely are both overly simplistic extremes.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#84

Post by fragmede »

belter wrote: post #82
fragmede wrote: post #79

Earlier quoted context omitted.

Why is that the bar though? Imagine LLMs as a kid that has a box of lego with a hundred million blocks in it, and it can assemble those blocks into any configuration possible. Is the fact that the kid doesn't have access to ABS plastic pellets and a molding machine, and so they can't make new pieces; does that really make us think that the kid just interpolates and can't create?

Actually yes...If the kid spends their whole life in the box and never invents a new block, that’s just combinatorics. We don’t call a chess engine ‘creative’ for finding novel moves, because we understand the rules. LLMs have rules too, they’re called weights. I want LLMs to create, but so far, every creative output I’ve seen is just a clever remix of training data. The most advanced models still fail a simple test:…

The combinatorics on choosing 500 pieces (words) out of a bag of 1.8 million pieces (approx parameters per layer for GPT-3) with replacement, and order matters works out to be something like 10^4600. Maybe you can't call that creativity, but you've got to admit that's a pretty big number.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#85

Post by danielmarkbruce »

SJC_Hacker wrote: post #59
danielmarkbruce wrote: post #57

Earlier quoted context omitted.

If you need basically rock solid evidence of X before you stop saying "this thing cannot do X", then you shouldn't be running a forward looking lab. There are only so many directions you can take, only so many resources at your disposal. Your intuition has to be really freakishly good to be running such a lab. He's done a lot of amazing work, but his stance on LLMs seems continuously off the mark.

The list of great minds who thought that "new fangled thing is nonsense" and later turned out to be horribly wrong is quite long and distinguished

*formerly great minds.

In many cases the folks in question were waaaaay past their best days.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#86

Post by mvdtnz »

antirez wrote: post #41
jxjnskkzxxhx wrote: post #34

Earlier quoted context omitted.

I don't see the contradiction between "stochastic parrot" and "strong summarisation abilities". Where I'm skeptical of LLM skepticism is that people use the term "stochastic parrot" disparagingly, as if they're not impressed. LLMs are stochastic parrots in the sense that they probabilistically guess sequences of things, but isn't it interesting how far that takes you already? I'd never have guessed. Fundamentally I q…

LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). So they don't learn probabilities, they learn how to continue the sentence with a given token. We apply softmax at the logits for mathematical reasons, and it is natural/simpler to think in terms of probabilities, but that's not what happens, nor the neural networks they are composed of is…

Just for anyone reading this who isn't sure, much like an LLM this is confident-sounding nonsense.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#87

Post by ksec »

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

>Btw, other researchers that were in the LeCun side, changed side recently, saying that now "is different" because of CoT, that is the symbolic reasoning they were blabling before. But CoT is still regressive next token without any architectural change, so, no, they were wrong, too.

Sorry I am a little lost reading the last part about regressive next token and it is still wrong. Could someone explain a little bit? Edit: Explained here further down the thread. ( https://news.ycombinator.com/item?id=43594813 )

I personally went from AI skeptic ( it wont ever replace all human, at least not in the next 10 - 20 years ) to AI scary simply because of the reasoning capability it gained. It is not perfect, far from it but I can immediately infer how both algorithm improvements and hardware advance could bring us in 5 years. And that is not including any new breakthrough.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#88

Post by csdvrx »

gibsonf1 wrote: post #76
csdvrx wrote: post #20

Earlier quoted context omitted.

A more generous take on the previous post is that the dominant paradigm of Math (consistent logic, which depends on many things like transitive preference) is wrong, and that another type of Math could work. If you look at the slide, the subtree of correct answers exists, what's missing is just a way to make them more prevalent instead of less. Personally, I think LeCun is just leaping to the wrong conclusion because…

My point is no type of math will work to model reason. Math is one of the many tools of reason, it is not the basis for reason. This is a very common error.

> My point is no type of math will work to model reason

Then I disagree with you.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#89

Post by mbesto »

antirez wrote: post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

I wanna believe everything you say (because you generally are a credible person) but a few things don't add up:

1. Weakest ever LLM? This one is really making me scratch my head. For a period of time Llama was considered to THE best. Furthermore, it's the third most used on OpenRouter (in the past month): https://openrouter.ai/rankings?view=month

2. Ignoring DeepSeek for a moment, Llama 2 and 3 require a special license from Meta if the products or services using the models have more than 700 million monthly active users. OpenAI, Claude and Gemini are not only closed source, but require a license/subscription to even get started.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#90

Post by ksec »

klipt wrote: post #55
antirez wrote: post #41

Earlier quoted context omitted.

LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). So they don't learn probabilities, they learn how to continue the sentence with a given token. We apply softmax at the logits for mathematical reasons, and it is natural/simpler to think in terms of probabilities, but that's not what happens, nor the neural networks they are composed of is…

> LLMs learn from examples where the logits are not probabilities, but how a given sentence continues (only one token is set to 1). But enough data implies probabilities. Consider 2 sentences: "For breakfast I had oats" "For breakfast I had eggs" Training on this data, how do you complete "For breakfast I had..."? There is no best deterministic answer. The best answer is a 50/50 probability distribution over "oats" a…

So it is still largely, probabilities pattern matching?
Post reply on HN