Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#51

Post by 317070 »

hnuser123456 wrote: post #38
hnfong wrote: post #18

I'm not an insider and I'm not sure whether this is directly related to "energy minimization", but "diffusion language models" have apparently gained some popularity in recent weeks. https://arxiv.org/abs/2502.09992 https://www.inceptionlabs.ai/news (these are results from two different teams/orgs) It sounds kind of like what you're describing, and nobody else has mentioned it yet, so take a look and see whether it's…

And they seem to be about 10x as fast as similar sized transformers.

No, 10x less sampling steps. Whether or not that means 10x faster remains to be seen, as a diffusion step tends to be more expensive than an autoregressive step.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#52

Post by flawn »

tyronehed wrote: post #26

Any transformer based LLM will never achieve AGI because it's only trying to pick the next word. You need a larger amount of planning to achieve AGI. Also, the characteristics of LLMs do not resemble any existing intelligence that we know of. Does a baby require 2 years of statistical analysis to become useful? No. Transformer architectures are parlor tricks. They are glorified Google but they're not doing anything o…

It's not about just picking the next word here, that doesn't at all refuse whether Transformers can achieve AGI. Words are just one representation of information. And whether it resembles any intelligence we know is also not an argument because there is no reason to believe that all intelligence is based on anything we've seen (e.g us, or other animals). The underlying architecture of Attention & MLPs can surely still depict something which we could call an AGI, and in certain tasks it surely can be considered an AGI already. I also don't know for certain whether we will hit any roadblocks or architectural asymptotes but I haven't come across any well-founded argument that Transformers definitely could not reach AGI.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#53

Post by josh-sematic »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

I don’t buy Lecun’s argument. Once you get good RL going (as we are now seeing with reasoning models) you can give the model a reward function that rewards a correct answer most highly, an “I’m sorry but I don’t know” less highly than that, a wrong answer penalized, a confidently wrong answer more severely penalized. As the RL learns to maximize rewards I would think it would find the strategy of saying it doesn’t know in cases where it can’t find an answer it deems to have a high probability of correctness.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#54

Post by rglover »

Not an ML researcher, but implementing these systems has shown this opinion to be correct. The non-determinism of LLMs is a feature, not a bug that can be fixed.

As a result, you'll never be able to get 100% consistent outputs or behavior (like you hypothetically can with a traditional algorithm/business logic). And that has proven out in usage across every model I've worked with.

There's also an upper-bound problem in terms of context where every LLM hits some arbitrary amount of context that causes it to "lose focus" and develop a sort of LLM ADD. This is when hallucinations and random, unrequested changes get made and a previously productive chat spirals to the point where you have to start over.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#55

Post by littlestymaar »

Lerc wrote: post #50
seanhunter wrote: post #9

Earlier quoted context omitted.

> The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. I don't think this explanation is correct. The input to the decoder at the end of all the attention heads etc (as I understand it) is a probability distribution over tokens. So the model as a whole…

I feel like we're stacking naive misinterpretations of how LLMs function on top of one another here. Grasping gradient descent and autoregressive generation can give you a false sense of confidence. It is like knowing how transistors make up logic gates and believing you know more than CPU design than you actually do. Rather than inferring from how you imagine the architecture working, you can look at examples and co…

No an ML researcher or anything (I'm basically only a few Karpathy video into ML, so please someone correct me if I'm misunderstanding this), but it seems that you're getting this backwards:

> One misconception is that predicting the next word means there is no internal idea on the word after next. The simple disproof of this is that models put 'an' instead of 'a' ahead of words beginning with vowels.

My understanding is that there's simply not “'an' ahead of a word that starts with a vowel”, the model (or more accurately, the sampler) picks “an” and then the model will never predict a word that starts with a consonant after that. It's not like it “knows” in advance that it wants to put a word with a vowel and then anticipates that it needs to put “an”, it generates a probability for both tokens “a” and “an”, picks one, and then when it generates the following token, it will necessarily take its previous choice into account and never puts a word starting with a vowel after it has already chosen “a”.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#56

Post by dtnewman »

TrainedMonkey wrote: post #5

This is a somewhat nihilistic take with an optimistic ending. I believe humans will never fix hallucinations. Amount of totally or partially untrue statements people make is significant. Especially in tech, it's rare for people to admit that they do not know something. And yet, despite all of that the progress keeps marching forward and maybe even accelerating.

I’m not sure I follow. Sure, people lie, and make stuff up all the time. If an LLM goes and parrots that, then I would argue that it isn’t hallucinating. Hallucinating would be where it makes something up that is not in its training site nor logically deducible from it.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#57

Post by Tryk »

josh-sematic wrote: post #53
ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

I don’t buy Lecun’s argument. Once you get good RL going (as we are now seeing with reasoning models) you can give the model a reward function that rewards a correct answer most highly, an “I’m sorry but I don’t know” less highly than that, a wrong answer penalized, a confidently wrong answer more severely penalized. As the RL learns to maximize rewards I would think it would find the strategy of saying it doesn’t kn…

How do you define the "correct" answer?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#58

Post by littlestymaar »

317070 wrote: post #51
hnuser123456 wrote: post #38

Earlier quoted context omitted.

And they seem to be about 10x as fast as similar sized transformers.

No, 10x less sampling steps. Whether or not that means 10x faster remains to be seen, as a diffusion step tends to be more expensive than an autoregressive step.

If I understood correctly, in practice they show actual speed improvement on high-end cards, because autoregressive LLMs are bandwidth limited and do not compute bound, so switching to a more expensive but less memory bandwidth heavy is going to work well on current hardware.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#59

Post by thijson »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

I watched an Andrej Karpathy video recently. He said that hallucination was because in the training data there were no examples where the answer is, "I don't know". Maybe I'm misinterpreting what he was saying though.

https://www.youtube.com/watch?v=7xTGNNLPyMI&t=4832s

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#60

Post by TZubiri »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

If multiple answers are equally likely, couldn't that be considered uncertainty? Conversely if there's only one answer and there's a huge leap to the second best, that's pretty certain.
Post reply on HN