Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#11

Post by Matthyze »

jawiggins wrote: post #4

I'm not an ML researcher, but I do work in the field. My mental model of AI advancements is that of a step function with s-curves in each step [1]. Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. Examples of steps…

> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found.

That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#12

Post by blueyes »

Sincere question - why doesn't RL-based fine-tuning on top of LLMs solve this or at least push accuracy above a minimum acceptable threshhold in many use cases? OAI has a team doing this for enterprise clients. Several startups rolling out of current YC batch are doing versions of this.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#13

Post by fhd2 »

ALittleLight wrote: post #7

I've never understood this critique. Models have the capability to say: "oh, I made a mistake here, let me change this" and that solves the issue, right? A little bit of engineering and fine tuning - you could imagine a model producing a sequence of statements, and reflecting on the sequence - updating things like "statement 7, modify: xzy to xyz"

I get "oh, I made a mistake" quite frequently. Often enough, it's just another hallucination, just because I contested the result, or even just prompted "double check this". Statistically speaking, when someone in a conversation says this, the other party is likely to change their position, so that's what an LLM does, too, replicating a statistically plausible conversation. That often goes in circles, not getting anywhere near a better answer.

Not an ML researcher, so I can't explain it. But I get a pretty clear sense that it's an inherent problem and don't see how it could be trained away.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#14

Post by __rito__ »

Sligtly related: Energy Based Models (EBMs) are better in theory and yet too resource intensive. I tried to sell using EBMs to my org, but the price for even a small use case was prohibitive.

I learned it from: https://youtube.com/playlist?list=PLLHTzKZzVU9eaEyErdV26ikyo...

Yann LeCun, and Michael Bronstein and his colleagues have some similarities in trying to properly Sciencify Deep Learning.

Yann LeCun's approach, as least for Vision has one core tenet- energy minimization, just like in Physics. In his course, he also shows some current arch/algos to be special cases for EBMs.

Yann believes that understanding the Whys of the behavior of DL algorithms are going to be beneficial in the long term rather than playing around with hyper-params.

There is also a case for language being too low-dimensional to lead to AGI even if it is solved. Like, in a recent video, he said that the total amount of data existing on all digitized books and internet are the same as what a human children takes in in the first 4/5 years. He considers this low.

There are also epistemological arguments against language not being able to lead to AGI, but I haven't heard him talk about them.

He also believes that Vision is a more important aspect of intellgence. One reason being it being very high-dim. (Edit) Consider an example. Take 4 monochrome pixels. All pixels can range from 0 to 255. 4 pixels can create 256^4 = 2^32 combinations. 4 words can create 4! = 24 combinations. Solving language is easier and therefore low-stakes. Remember the monkey producing a Shakespeare play by randomly punching typewriter keys? If that was an astronomically big number, think how obscenely long it would take a monkey to paint Mona Lisa by randomly assigning pixel values. Left as an exercise to the reader.

Juergen Schmidhuber has gone a lot queit now. But he also told that a world-model, explicitly included in training is reasoning is better, rather than only text or image or whatever. He has a good paper with Lucas Beyer.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#15

Post by semi-extrinsic »

Matthyze wrote: post #11
jawiggins wrote: post #4

I'm not an ML researcher, but I do work in the field. My mental model of AI advancements is that of a step function with s-curves in each step [1]. Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. Examples of steps…

> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.

Punctuated equilibrium theory.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#16

Post by estebarb »

seanhunter wrote: post #9
ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

> The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. I don't think this explanation is correct. The input to the decoder at the end of all the attention heads etc (as I understand it) is a probability distribution over tokens. So the model as a whole…

The problem is exactly that: the probability distribution. The network has no way to say: 0% everyone, this is non sense, backtrack everything.

Other architectures, like energy based models or bayesian ones can assess uncertainty. Transformers simply cannot do it (yet). Yes, there are ways to do it, but we are already spending millions to get coherent phrases, few ones will burn billions to train a model that can do that kind of assessments.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#17

Post by duskwuff »

seanhunter wrote: post #9
ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

> The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. I don't think this explanation is correct. The input to the decoder at the end of all the attention heads etc (as I understand it) is a probability distribution over tokens. So the model as a whole…

Right. And, as a result, low token-level confidence can end up indicating "there are other ways this could have been worded" or "there are other topics which could have been mentioned here" just as often as it does "this output is factually incorrect". Possibly even more often, in fact.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#18

Post by hnfong »

I'm not an insider and I'm not sure whether this is directly related to "energy minimization", but "diffusion language models" have apparently gained some popularity in recent weeks.

https://arxiv.org/abs/2502.09992

https://www.inceptionlabs.ai/news

(these are results from two different teams/orgs)

It sounds kind of like what you're describing, and nobody else has mentioned it yet, so take a look and see whether it's relevant.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#19

Post by rscho »

ALittleLight wrote: post #7

I've never understood this critique. Models have the capability to say: "oh, I made a mistake here, let me change this" and that solves the issue, right? A little bit of engineering and fine tuning - you could imagine a model producing a sequence of statements, and reflecting on the sequence - updating things like "statement 7, modify: xzy to xyz"

"Oh, I emptied your bank account here, let me change this."

For AI to really replace most workers like some people would like to see, there are plenty of situations where hallucinations are a complete no-go and need fixing.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#20

Post by ortsa »

estebarb wrote: post #16
seanhunter wrote: post #9

Earlier quoted context omitted.

> The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. I don't think this explanation is correct. The input to the decoder at the end of all the attention heads etc (as I understand it) is a probability distribution over tokens. So the model as a whole…

The problem is exactly that: the probability distribution. The network has no way to say: 0% everyone, this is non sense, backtrack everything. Other architectures, like energy based models or bayesian ones can assess uncertainty. Transformers simply cannot do it (yet). Yes, there are ways to do it, but we are already spending millions to get coherent phrases, few ones will burn billions to train a model that can do…

Has anybody ever messed with adding a "backspace" token?
Post reply on HN