I don't think it's a coincidence that he is interested in non-LLM solutions, since he mentioned last year on Twitter that he doesn't have an internal monologue (I hope this is not taken as disparaging of him in any way). His criticisms of LLMs never made sense, and the success of reasoning models has shown him to be definitely wrong.
Ask HN: Any insider takes on Yann LeCun's push against current architectures?
191–200 of 343 posts
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#192You could reframe the way LLMs are currently trained as energy minimization, since the Boltzmann distribution that links physics and information theory (and correspondingly, probability theory as well) is general enough to include all standard loss functions as special cases. It's also pretty straightforward to include RL in that category as well.
I think what Lecun is probably getting at is that there's currently no way for a model to say "I don't know". Instead, it'll just do its best. For esoteric topics, this can result in hallucinations; for topics where you push just past the edge of well-known and easy-to-Google, you might get a vacuously correct response (i.e. repetition of correct but otherwise known or useless information). The models are trained to output a response that meets the criteria of quality as judged by a human, but there's no decent measure (that I'm aware of) of the accuracy of the knowledge content, or the model's own limitations. I actually think this is why programming and mathematical tasks have such a large impact on model performance: because they encode information about correctness directly into the task.
So Yann is probably right, though I don't know that energy minimization is a special distinction that needs to be added. Any technique that we use for this task could almost certainly be framed as energy minimization of some energy function.
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#193Earlier quoted context omitted.
No. > The sun feels hot on your skin. No matter how many times you read that, you cannot understand what the experience is like. > You can read a book about Yoga and read about the Tittibhasana pose But by just reading you will not understand what it feels like. And unless you are in great shape and with greate balance you will fail for a while before you get it right. (which is only human). I have read what shooting…
> No. Huh, text definitely encodes multimodal experiences, it's just not as accurate and as rich encoding as the encodings of real sensations.
Text describes semantic space. Not everything maps to semantic space losslessly.
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#194Why do they have to use the word “hallucination” when the model makes a mistake, if you tell your teacher or boss you didn’t get the answer wrong, you’ve hallucinated it, he will send you to the hospital.
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#195Earlier quoted context omitted.
Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentia…
> Doesn't Language itself encode multimodal experiences Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we don't have great text datasets to describe many things in enough detail, but there isn't any reason we couldn't.
There are absolutely reasons that we cannot capture the entirety—or even a proper image—of human cognition in semantic space.
Cognition is not purely semantic. It is dynamic, embodied, socially distributed, culturally extended, and conscious.
LLMs are great semantic heuristic machines. But they don't even have access to those other components.
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#196Earlier quoted context omitted.
Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentia…
No. > The sun feels hot on your skin. No matter how many times you read that, you cannot understand what the experience is like. > You can read a book about Yoga and read about the Tittibhasana pose But by just reading you will not understand what it feels like. And unless you are in great shape and with greate balance you will fail for a while before you get it right. (which is only human). I have read what shooting…
Doesn't this imply that the future of AGI lies not just in vision and text but in tactile feelings and actions as well ?
Essentially, engineering the complete human body and mind including the nervous system. Seems highly intractable for the next couple of decades at least.
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#197Earlier quoted context omitted.
Interesting but a bit non-sequitur. Humans learn and get things wrong. A formative mind is a seperate subject. But a 3 year old is vastly intelligent vs an LLM. Comparing the sounds from a 3 year old and the binary tokens from an LLM is simply indulging the illusion. I am also not convinced that magicians saw people in half, and thise people survive, defying medical and physical science.
I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM" Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN. What in the world doe…
> What in the world does it mean that a 3 year old is smarter than an LLM?
Because LLMs have terrible comprehension of the real world. Here's an example:
> You: If you put a toddler next to a wig on the floor, which reaches higher?
> ChatGPT: The wig would typically reach higher than the toddler, especially if the wig is a standard size or has long hair. Toddlers are generally around 2 to 3 feet tall, while wigs can range in size, but many wigs are designed to be worn by adults and have hair that extends well beyond a toddler's height. So, unless the wig is very small or short, it would likely reach higher.
Both the toddler and the wig would be lying down (unless the toddler can stand already), with the wig likely collapsing into a very flat heap, making the toddler "reach higher". Since it lacks comprehension of how these things behave in the real world, ChatGPT goes on about length of the wig and the average height of toddlers, which are both qualities that are not even relevant.
If you could communicate this problem to a toddler, they would probably intuit it even with their still limited understanding of the world.
Here's another fun prompt to mess with LLMs:
> You: I'm trying to make a rope, made of only aubergines, not using any twine or string. What kind of knot would you recommend to tie together each aubergine with the next? I'm trying to use up as little of the length of each aubergine as possible.
It's obviously impossible nonsense if you ask any human, but ChatGPT will happily recommend you some techniques for tying aubergines in a knot.
And if you really want a laugh follow up with:
> You: Are they the worst fruit for this?
> ChatGPT: Aubergines (or eggplants) aren't ideal for making a rope, but they might not be the absolute worst, either. [...] Potentially Better Fruits: Apples or Pears: These fruits are firmer and have a smoother, more uniform shape, making them better suited for knotting.
Thanks to the large corpus of text ChatGPT is trained on, it's really good at hiding that it has basically no comprehension of the real world, however if you go unconventional places the illusion quickly breaks.
Bonus round:
> You: You're clueless about fruit!
> ChatGPT: Haha, fair enough! Looks like I missed the mark there. You’re probably right, though—there are definitely better fruits for this kind of thing than I suggested. Let’s try again. If you're looking to make a rope from a fruit specifically (and not a vegetable or tuber), something like coconuts or durian would be way better options, since they’re harder, more rigid, and can handle some tying pressure without collapsing. And apples and pears would definitely hold up well due to their firmer texture.
Coconuts!
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#198Earlier quoted context omitted.
I'm reminded of the story of Helen Keller, and how it took a long time for her to realize that the symbols her teacher was signing into her hand had meaning, as she was blind and deaf and only experienced the world via touch and smell. She didn't get it until her teacher spelled the word "water" as water from a pump was flowing over her hand. In other words, a multimodal experience. If the model only sees text, it ca…
It is important to note that Helen Keller was not born blind and deaf, though. (I am not reducing the struggle she went through. Just commentary on embodied cognition and learning.) There were around 19 months of normal speech and hearing development until then and also 3D object space traversal and object manipulation.
[deleted]
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#199Earlier quoted context omitted.
I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM" Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN. What in the world doe…
> What in the world does it mean that a 3 year old is smarter than an LLM? Because LLMs have terrible comprehension of the real world. Here's an example: > You: If you put a toddler next to a wig on the floor, which reaches higher? > ChatGPT: The wig would typically reach higher than the toddler, especially if the wig is a standard size or has long hair. Toddlers are generally around 2 to 3 feet tall, while wigs can…
> Because LLMs have terrible comprehension of the real world.
That doesn't seem to be the case.
> You: If you put a toddler next to a wig on the floor, which reaches higher? > ChatGPT: ...
I answered it wrong too.
I had to read it, and your reaction to the implied obvious reasoning 3 times, to figure out the implied obvious reasoning, and understand your intent was the toddler was standing and the wig was laying in a heap.
I scored 99.9+% on the SAT and LSAT. I think that implies this isn't some reasoning deficit, lack of familiarity with logical reasoning on my end, or lack of rigor in reasoning.
I have no particular interest in this argument. I think that implies that I'm not deploying motivated reasoning, i.e. it discounts the possibility that I may have experienced it as confusion that required re-reading the entire comment 3 times, but perhaps I had subconcious priors.
Would a toddler even understand the question? (serious question, I'm not familiar with 3 year olds)
Does this shed any light on how we'd work an argument along the lines of our deaf and mute friend typing?
Edit: you edited in some more examples, I found it's aubergine answers quite clever! (Ex. notching). I can't parse out a convincing argument this is somehow less knowledge than a 3 year old -- it's giving better answers than me that are physical! I thought youd be sharing it asserting obviously nonphysical answers
Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?
#200I don't think it's a coincidence that he is interested in non-LLM solutions, since he mentioned last year on Twitter that he doesn't have an internal monologue (I hope this is not taken as disparaging of him in any way). His criticisms of LLMs never made sense, and the success of reasoning models has shown him to be definitely wrong.
Yes, it is fascinating that humans can have such seemingly fundamental differences in how they function 'under the hood.' I also have a friend who is highly intelligent—they earned a STEM PhD from one of the best universities in the world—yet they struggle to follow complex movie plots, despite having a photographic memory. It would be interesting to develop mirror LLMs (or Large Anything Models) for all these different types of brains so we can study how exactly these traits manifest and interact.