Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#61

Post by janalsncm »

throw310822 wrote: post #40
janalsncm wrote: post #39

I am an MLE not an expert. However, it is a fundamental problem that our current paradigm of training larger and larger LLMs cannot ever scale to the precision people require for many tasks. Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. https://arxiv.org/pdf/2402.04494

> Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. This is true also for the much bigger neural net that works in your brain, and even if you're the world champion of chess. Clearly your argument doesn't hold water.

For the sake of argument let’s say an artificial neural net is approximately the same as the brain. It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net. So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#62

Post by jpadkins »

Tryk wrote: post #57
josh-sematic wrote: post #53

Earlier quoted context omitted.

I don’t buy Lecun’s argument. Once you get good RL going (as we are now seeing with reasoning models) you can give the model a reward function that rewards a correct answer most highly, an “I’m sorry but I don’t know” less highly than that, a wrong answer penalized, a confidently wrong answer more severely penalized. As the RL learns to maximize rewards I would think it would find the strategy of saying it doesn’t kn…

How do you define the "correct" answer?

obviously the truth is what is the most popular. /s

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#63

Post by visarga »

tyronehed wrote: post #26

Any transformer based LLM will never achieve AGI because it's only trying to pick the next word. You need a larger amount of planning to achieve AGI. Also, the characteristics of LLMs do not resemble any existing intelligence that we know of. Does a baby require 2 years of statistical analysis to become useful? No. Transformer architectures are parlor tricks. They are glorified Google but they're not doing anything o…

The transformer is a simple and general architecture. Being such a flexible model, it needs to learn "priors" from data, it makes few assumptions on its distribution from the start. The same architecture can predict protein folding and fluid dynamics. It's not specific to language.

We on the other hand are shaped by billions of years of genetic evolution, and 200k years of cultural evolution. If you count the total number of words spoken by 110 billion people who ever lived, assuming 1B estimated words per human during their lifetime, it comes out to 10 million times the size of GPT-4's training set.

So we spent 10 million more words discovering than it takes the transformer to catch up. GPT-4 used 10 thousand people's worth of language to catch up all that evolutionary finetuning.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#64

Post by unsupp0rted »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

> The problem with LLMs is that the output is inherently stochastic

Isn't that true with humans too?

There's some leap humans make, even as stochastic parrots, that lets us generate new knowledge.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#65

Post by spmurrayzzz »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

> i.e there isn't a "I don't have enough information" option.

This is true in terms of default mode for LLMs, but there's a fair amount of research dedicated to the idea of training models to signal when they need grounding.

SelfRAG is an interesting, early example of this [1]. The basic idea is that the model is trained to first decide whether retrieval/grounding is necessary and then, if so, after retrieval it outputs certain "reflection" tokens to decide whether a passage is relevant to answer a user query, whether the passage is supported (or requires further grounding), and whether the passage is useful. A score is calculated from the reflection tokens.

The model then critiques itself further by generating a tree of candidate responses, and scoring them using a weighted sum of the score and the log probabilities of the generated candidate tokens.

We can probably quibble about the loaded terms used here like "self-reflection", but the idea that models can be trained to know when they don't have enough information isn't pure fantasy today.

[1] https://arxiv.org/abs/2310.11511

EDIT: I should also note that I generally do side with Lecun's stance on this, but not due to the "not enough information" canard. I think models learning from abstraction (i.e. JEPA, energy-based models) rather than memorization is the better path forward.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#66

Post by EEgads »

Yann LeCun understands this is an electrical engineering and physical statistics of machine problem and not a code problem.

The physics of human consciousness are not implemented in a leaky symbolic abstraction but the raw physics of existence.

The sort of autonomous system we imagine when thinking AGI must be built directly into substrate and exhibit autonomous behavior out of the box. Our computers are blackboxes made in a lab without centuries of evolving in the analog world, finding a balance to build on. They either can do a task or cannot. Obviously from just looking at one we know how few real world tasks it can just get up and do.

Code isn’t magic, it’s instruction to create a machine state. There’s no inherent intelligence to our symbolic logic. It’s an artifact of intelligence. It cannot imbue intelligence into a machine.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#67

Post by janalsncm »

thewarrior wrote: post #42
janalsncm wrote: post #39

I am an MLE not an expert. However, it is a fundamental problem that our current paradigm of training larger and larger LLMs cannot ever scale to the precision people require for many tasks. Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. https://arxiv.org/pdf/2402.04494

Any chance that “reasoning” can fix this

It kind of depends. You can broadly call any kind of search “reasoning”. But search requires 1) enumerating your possible options and 2) assigning some value to those options. Real world problem solving makes both of those extremely difficult.

Unlike in chess, there’s a functionally infinite number of actions you can take in real life. So just argmax over possible actions is going to be hard.

Two, you have to have some value function of how good an action is in order to argmax. But many actions are impossible to know the value of in practice because of hidden information and the chaotic nature of the world (butterfly effect).

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#68

Post by kjrfghslkdjfl »

ActorNightly wrote: post #2

Not an official ML researcher, but I do happen to understand this stuff. The problem with LLMs is that the output is inherently stochastic - i.e there isn't a "I don't have enough information" option. This is due to the fact that LLMs are basically just giant look up maps with interpolation. Energy minimization is more of an abstract approach to where you can use architectures that don't rely on things like different…

[dead]

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#69

Post by unsupp0rted »

tyronehed wrote: post #26

Any transformer based LLM will never achieve AGI because it's only trying to pick the next word. You need a larger amount of planning to achieve AGI. Also, the characteristics of LLMs do not resemble any existing intelligence that we know of. Does a baby require 2 years of statistical analysis to become useful? No. Transformer architectures are parlor tricks. They are glorified Google but they're not doing anything o…

> Does a baby require 2 years of statistical analysis to become useful?

Well yes, actually.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#70

Post by jpadkins »

janalsncm wrote: post #61
throw310822 wrote: post #40

Earlier quoted context omitted.

> Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. This is true also for the much bigger neural net that works in your brain, and even if you're the world champion of chess. Clearly your argument doesn't hold water.

For the sake of argument let’s say an artificial neural net is approximately the same as the brain. It sounds like you agree with me that smaller programs are both more efficient and more effective than a larger neural net. So you should also agree with me that those who say the only path to AGI is LLM maximalism are misguided.

smaller programs are better than artificial or organic neural net for constrained problems like chess. But chess programs don't generalize to any other intelligence applications, like how organic neural nets do today.
Post reply on HN