Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#71

Post by bobosha »

I argue that JEPA and its Energy-Based Model (EBM) framework fail to capture the deeply intertwined nature of learning and prediction in the human brain—the “yin and yang” of intelligence. Contemporary machine learning approaches remain heavily reliant on resource-intensive, front-loaded training phases. I advocate for a paradigm shift toward seamlessly integrating training and prediction, aligning with the principles of online learning.

Disclosure: I am the author of this paper.

Reference: (PDF) Hydra: Enhancing Machine Learning with a Multi-head Predictions Architecture. Available from: https://www.researchgate.net/publication/381009719_Hydra_Enh... [accessed Mar 14, 2025].

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#72

Post by vhantz »

killthebuddha wrote: post #10

I've always felt like the argument is super flimsy because "of course we can _in theory_ do error correction". I've never seen even a semi-rigorous argument that error correction is _theoretically_ impossible. Do you have a link to somewhere where such an argument is made?

> of course we can _in theory_ do error correction

Oh yeah? This is begging the question.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#73

Post by yunwal »

littlestymaar wrote: post #55
Lerc wrote: post #50

Earlier quoted context omitted.

I feel like we're stacking naive misinterpretations of how LLMs function on top of one another here. Grasping gradient descent and autoregressive generation can give you a false sense of confidence. It is like knowing how transistors make up logic gates and believing you know more than CPU design than you actually do. Rather than inferring from how you imagine the architecture working, you can look at examples and co…

No an ML researcher or anything (I'm basically only a few Karpathy video into ML, so please someone correct me if I'm misunderstanding this), but it seems that you're getting this backwards: > One misconception is that predicting the next word means there is no internal idea on the word after next. The simple disproof of this is that models put 'an' instead of 'a' ahead of words beginning with vowels. My understandin…

The model still has some representation of whether the word after an/a is more likely to start with a vowel or not when it outputs a/an. You can trivially understand this is true by asking LLMs to answer questions with only one correct answer.

"The animal most similar to a crocodile is:"

https://chatgpt.com/share/67d493c2-f28c-8010-82f7-0b60117ab2...

It will always say "an alligator". It chooses "an" because somewhere in the next word predictor it has already figured out that it wants to say alligator when it chooses "an".

If you ask the question the other way around, it will always answer "a crocodile" for the same reason.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#74

Post by throw310822 »

janalsncm wrote: post #61
throw310822 wrote: post #40

Earlier quoted context omitted.

> Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. This is true also for the much bigger neural net that works in your brain, and even if you're the world champion of chess. Clearly your argument doesn't hold water.

For the sake of argument let’s say an artificial neural net is approximately the same as the brain. It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net. So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

> It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net.

At playing chess. (But also at doing sums and multiplications, yay!)

> So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

No. First of all, it's a claim you just made up. What we're talking about is people saying that LLMs are not the path to AGI- an entirely different claim.

Second, assuming there's any coherence to your argument, the fact that a small program can outclass an enormous NN is irrelevant to the question of whether the enormous NN is the right way to achieve AGI: we are "general intelligences" and we are defeated by the same chess program. Unless you mean that achieving the intelligence of the greatest geniuses that ever lived is still not enough.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#75

Post by inimino »

I have a paper coming up that I modestly hope will clarify some of this.

The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#76

Post by jedberg »

inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

> and biologically implausible

I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains.

Is this a common thing in ML papers or something you came up with?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#77

Post by vhantz »

bitwize wrote: post #37

Ever hear of Dissociated Press? If not, try the following demonstration. Fire up Emacs and open a text file containing a lot of human-readable text. Something off Project Gutenberg, say. Then say M-x dissociated-press and watch it spew hilarious, quasi-linguistic garbage into a buffer for as long as you like. Dissociated Press is a language model. A primitive, stone-knives-and-bearskins language model, but a language…

+1

But there's a fundamental difference between Markov chains and transformers that should be noted. Markov chains only learn how likely it is for one token to follow another. Transformers learn how likely it is for a set of token to be seen together. Transformers add a wider context to msrkov chain. That quantitative change leads to a qualitative improvement: transformers generate text that is semantically plausible.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#78

Post by esafak »

jedberg wrote: post #76
inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

> and biologically implausible I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains. Is this a common thing in ML papers or something you came up with?

Evolution does not need to converge on the optimum solution.

Have you heard of https://en.wikipedia.org/wiki/Bio-inspired_computing ?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#79

Post by littlestymaar »

yunwal wrote: post #73
littlestymaar wrote: post #55

Earlier quoted context omitted.

No an ML researcher or anything (I'm basically only a few Karpathy video into ML, so please someone correct me if I'm misunderstanding this), but it seems that you're getting this backwards: > One misconception is that predicting the next word means there is no internal idea on the word after next. The simple disproof of this is that models put 'an' instead of 'a' ahead of words beginning with vowels. My understandin…

The model still has some representation of whether the word after an/a is more likely to start with a vowel or not when it outputs a/an. You can trivially understand this is true by asking LLMs to answer questions with only one correct answer. "The animal most similar to a crocodile is:" https://chatgpt.com/share/67d493c2-f28c-8010-82f7-0b60117ab2... It will always say "an alligator". It chooses "an" because somewher…

Again, that's not a good example I think because everything about the answer is in the prompt, so obviously from the start the "alligator" is high, but then it's just waiting for an "an" to occur to have an occasion to put that.

That doesn't mean it knows "in advance" what it want to say, it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#80

Post by jedberg »

esafak wrote: post #78
jedberg wrote: post #76

Earlier quoted context omitted.

> and biologically implausible I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains. Is this a common thing in ML papers or something you came up with?

Evolution does not need to converge on the optimum solution. Have you heard of https://en.wikipedia.org/wiki/Bio-inspired_computing ?

It does not, you're right. But it's an interesting way to approach the problem never the less. And given that we definitely aren't as efficient as a human brain right now, it makes sense to look at the brain for inspiration.
Post reply on HN