Live data from Hacker News

Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

newsweek.com

111–120 of 147 posts

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#111
post #108
post #96

Earlier quoted context omitted.

I have not followed all of LeCun's past statements, but - if the "core belief" is that the LLM architecture cannot be the way to AGI, that is more of an "educated bet", which does not get falsified when LLMs improve but still suggest their initial faults. If seeing that LLMs seem constrained in the "reactive system" as opposed to a sought "deliberative system" (or others would say "intuitive" vs "procedural" etc.) wa…

If you say LLMs are a dead end, and you give a few examples of things they will never be able to do, and a few months later they do it, and you just respond by stating that sure they can do that but they're still a dead end and won't be able to do this. Rinse and repeat. After a while you question whether LLMs are actually a dead end

This is a normal routine topical in Epistemology in the perspective of Lakatos.

As I said, it will depend on whether the examples in question were actually substantial part of the "core belief".

For example: "But can they perform procedures?" // "Look at that now" // "But can they do it structurally? Consistently? Reliably?" // "Look at that now" // "But is that reasoning integrated or external?" // "Look at that now" // "But is their reasoning fully procedurally vetted?" (etc.)

I.e.: is the "progress" (which would be the "anomaly" in scientific prediction) part of the "substance" or part of the "form"?

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#112

Earlier quoted context omitted.

The list of great minds who thought that "new fangled thing is nonsense" and later turned out to be horribly wrong is quite long and distinguished

> Heavier-than-air flying machines are impossible. -Lord Kelvin. 1895 > I think there is a world market for maybe five computers. Thomas Watson, IBM. 1943 > On talking films: “They’ll never last.” -Charlie Chaplin. > This ‘telephone’ has too many shortcomings… -William Orton, Western Union. 1876 > Television won’t be able to hold any market -Darryl Zanuck, 20th Century Fox. 1946 > Louis Pasteur’s theory of germs is r…

In fairness to Irving Fisher: if you bought into the market at its peak in 1929, you wouldn't recover your original investment until about 1960.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#113
post #88

Earlier quoted context omitted.

My point is no type of math will work to model reason. Math is one of the many tools of reason, it is not the basis for reason. This is a very common error.

> My point is no type of math will work to model reason Then I disagree with you.

I'm ignorantly curious of what type of math will work in your view. Genuine question, I just want to be educated.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#114
post #21

outside of text generation and search, LLMs have not delivered any significant value

I personally have greatly benefitted from LLM's helping me reason about problems and make progress on many diverse issues across professional, recreational and mental health difficulties. I think that asking whether it's just "text generation and search" rather than something that transcends it is as meaningful as asking whether an airplane really "flies" or just "applies thrust and generates lift".

this is just a form of search

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#115
post #89

Earlier quoted context omitted.

I wanna believe everything you say (because you generally are a credible person) but a few things don't add up: 1. Weakest ever LLM? This one is really making me scratch my head. For a period of time Llama was considered to THE best. Furthermore, it's the third most used on OpenRouter (in the past month): https://openrouter.ai/rankings?view=month 2. Ignoring DeepSeek for a moment, Llama 2 and 3 require a special lice…

Doesn't OpenRouter ranking include pricing? Not really a good measure of quality or performance but of cost effectiveness

I mean it literally says on the page:

"Shown are the sum of prompt and completion tokens per model, normalized using the GPT-4 tokenizer."

Also, it ranks the use of Llama that is provided by cloud providers (for example, AWS Lamda).

I get that OpenRouter is imperfect but its a good proxy to objectively make a claim that an LLM is "the weakest ever"

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#116
post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

Everything is possible with math. Just ask string theorists.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#117
post #23

As LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between…

> there is no probabilistic link between the words of a text and the gist of the content

Using n-gram/skip-gram model over the long text you can predict probabilities of word pairs and/or word triples (effectively collocations [1]) in the summary.

[1] https://en.wikipedia.org/wiki/Collocation

Then, by using (beam search and) an n-gram/skip-gram model of summaries, you can generate the text of a summary, guided by preference of the words pairs/triples predicted by the first step.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#118

Earlier quoted context omitted.

I don't think the difference is material, between "they learn probabilities" Vs "they learn how they want a sentence to continue". Seems like an implementation detail to me. In fact, you can add a temperature, set it to zero, and you become deterministic, so no probabilities anywhere. The fact is, they learn from examples of sequences and are very good at finding patterns in those sequences, to a point that they "sou…

I don't find anything surprising about that. What humans generally see of each other is little more than outer shells that are made out of sequenced linguistic patterns. They generally find that completely sufficient. (All things considered, you may be right to be suspicious of me.)

Nah, to me you're just an average person on the internet. If the recent developments don't surprise you, I just chalk it up to lack of curiosity. I'm well aware that people like you exist, most people are like that in fact. My comment was referring to experts specifically.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#119
post #25

Earlier quoted context omitted.

Why is changing one’s mind when confronted with new evidence a negative signifier of reputation for you?

If you need basically rock solid evidence of X before you stop saying "this thing cannot do X", then you shouldn't be running a forward looking lab. There are only so many directions you can take, only so many resources at your disposal. Your intuition has to be really freakishly good to be running such a lab. He's done a lot of amazing work, but his stance on LLMs seems continuously off the mark.

I'm going to wear the tinfoil hat: a firm is able to produce a sought-after behavior a few months later and throws people off. Is it more likely that the firm (worth billions at this point) is engineering these solutions into the model, or is it because of emergent neural network architectural magic?

I'm not saying that they are being bad actors, just saying this is more probable in my mind than an LLM breakthrough.

Re: Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete

#120

Earlier quoted context omitted.

LLMs literally are just predicting tokens with a probabilistic model. They’re incredibly complicated and sophisticated models, but they still are just incredibly complicated and sophisticated models for predicting tokens. It’s maybe unexpected that such a thing can do summarization, but it demonstrably can.

The rub is that we don't know if intelligence is anything more than "just predicting next output".

I think we do.
Post reply on HN