Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#181

Post by spyder »

ThinkBeat wrote: post #146
codenlearn wrote: post #138

Earlier quoted context omitted.

Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentia…

No. > The sun feels hot on your skin. No matter how many times you read that, you cannot understand what the experience is like. > You can read a book about Yoga and read about the Tittibhasana pose But by just reading you will not understand what it feels like. And unless you are in great shape and with greate balance you will fail for a while before you get it right. (which is only human). I have read what shooting…

> No.

Huh, text definitely encodes multimodal experiences, it's just not as accurate and as rich encoding as the encodings of real sensations.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#182

Post by AlexCoventry »

littlestymaar wrote: post #58
317070 wrote: post #51

Earlier quoted context omitted.

No, 10x less sampling steps. Whether or not that means 10x faster remains to be seen, as a diffusion step tends to be more expensive than an autoregressive step.

If I understood correctly, in practice they show actual speed improvement on high-end cards, because autoregressive LLMs are bandwidth limited and do not compute bound, so switching to a more expensive but less memory bandwidth heavy is going to work well on current hardware.

The SEDD architecture [1] probably allows for parallel sampling of all tokens in a block at once, which may be faster but not necessarily less computationally demanding in terms of runtime times computational resources used.

[1] Which Inception Labs's new models may be based on; one of the cofounders is a co-author. See equations 18-20 in https://arxiv.org/abs/2310.16834

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#183

Post by AlexCoventry »

Not an insider, but:

I don't know about you, but I certainly don't generate text autoregressively, token by token. Also, pretty sure I don't learn by global updates based on taking the derivative of some objective function of my behavior with respect to every parameter defining my brain. So there's good biological reason to think we can go beyond the capabilities of current architectures.

I think probably an example of the kind of new architectures he supports is FB's Large Concept Models [1]. It's still a self-attention, autoregressive architecture, but the unit of regression is a sentence rather than a token. It maps sentences into a latent space via an autoencoder architecture, then has a transformer architecture in which the tokens are elements in that latent space.

[1] https://arxiv.org/abs/2412.08821

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#184

Post by bmitc »

not2b wrote: post #144
codenlearn wrote: post #138

Earlier quoted context omitted.

Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentia…

I'm reminded of the story of Helen Keller, and how it took a long time for her to realize that the symbols her teacher was signing into her hand had meaning, as she was blind and deaf and only experienced the world via touch and smell. She didn't get it until her teacher spelled the word "water" as water from a pump was flowing over her hand. In other words, a multimodal experience. If the model only sees text, it ca…

It is important to note that Helen Keller was not born blind and deaf, though. (I am not reducing the struggle she went through. Just commentary on embodied cognition and learning.) There were around 19 months of normal speech and hearing development until then and also 3D object space traversal and object manipulation.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#185

Post by bmitc »

spyder wrote: post #181
ThinkBeat wrote: post #146

Earlier quoted context omitted.

No. > The sun feels hot on your skin. No matter how many times you read that, you cannot understand what the experience is like. > You can read a book about Yoga and read about the Tittibhasana pose But by just reading you will not understand what it feels like. And unless you are in great shape and with greate balance you will fail for a while before you get it right. (which is only human). I have read what shooting…

> No. Huh, text definitely encodes multimodal experiences, it's just not as accurate and as rich encoding as the encodings of real sensations.

It's just a description, not an encoding.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#186

Post by zmmmmm »

So much of our fundamental scientific progress has been made by people who were considered crazy and their ideas delusional. Even mundane software engineering is done with layers of code review and automated tests because even the best engineers are still pretty bad at it. At a larger level, humanity itself seems to largely operate more like an ensemble method where many people in parallel solve problems and we empirically find who was "hallucinating".

Which is just to say, it feels to me like there's a danger that the stochastic nature of outputs is fundamental to true creative intelligence and all attempts to stamp it out will result in lower accuracy overall. Rather we should be treating it more like we do actual humans and expect errors and put layers of process around things where it matters to make them safe.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#187

Post by ambicapter »

spyder wrote: post #181
ThinkBeat wrote: post #146

Earlier quoted context omitted.

No. > The sun feels hot on your skin. No matter how many times you read that, you cannot understand what the experience is like. > You can read a book about Yoga and read about the Tittibhasana pose But by just reading you will not understand what it feels like. And unless you are in great shape and with greate balance you will fail for a while before you get it right. (which is only human). I have read what shooting…

> No. Huh, text definitely encodes multimodal experiences, it's just not as accurate and as rich encoding as the encodings of real sensations.

I don't think GP is asserting that the multimodal encoding is "more rich" or "more accurate", I think they are saying that the felt modality is a different thing than the text modality entirely, and that the former isn't contained in the latter.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#188

Post by eximius »

mft_ wrote: post #103
eximius wrote: post #95

I believe that so long as weights are fixed at inference time, we'll be at a dead end. Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm not sure. Ultimately, I think an architecture around "looping" where the model outputs are both some form of "self update" and "optional actionality" such that interacting with the model is more "sampling from a thought space" will be required.

Very much this. I’ve been wondering why I’ve not seen it much discussed.

Self updating requires learning to learn, which I'm not sure we know how to do.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#189

Post by littlestymaar »

metaxz wrote: post #114
littlestymaar wrote: post #79

Earlier quoted context omitted.

Again, that's not a good example I think because everything about the answer is in the prompt, so obviously from the start the "alligator" is high, but then it's just waiting for an "an" to occur to have an occasion to put that. That doesn't mean it knows "in advance" what it want to say, it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt.

You write: "it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt" - but isn't that the whole point: at the moment the model writes "an", it isn't just spitting out a random article (or a 50/50 distribution of articles or other words for that matter); rather, "an" gets a high probability because the model internally knows that "alligator" is the correct thing…

> While it can only emit one token in this step, it will emit "an" to make it consistent with its alligator knowledge "lurking".

It will also emit "a" from time to time without issue though, but will never spit "alligator" right after that, that's it.

> Sure, it derives from the prompt but so does every an LLM generates, and same for any other AI mechanism for generating answers.

Not really, because of the autoregressive nature of LLMs, the longer the response the more it will depend on its own response rather than the prompt. That's why you can see totally opposite response from LLM to the same query if you aren't asking basic factual questions. I saw a tool on reddit a few month ago that allowed you to see which words in the generation where the most “opinionated” (where the sampler had to chose between alternative words that were close in probability) and where it was easy to see that you could dramatically affect the result by just changing certain words.

> "an" gets a high probability because the model internally knows that "alligator" is the correct thing after that.

This is true, though it only works with this kind of prompt because the output of the LLM has little impact on the generation.

Globally I see what you mean, and I don't disagree with you, but at the same time, I think that saying that LLMs have a sense of anticipating the further token misses their ability to get driven astray by their own output: they have some information that will affect further tokens but any token that get spit can, and will, change that information in a way that can dramatically change the “plans”. And that's why I think using trivial questions isn't a good illustration, because it pushes this effect under the rug.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#190

Post by mountainriver »

cm2012 wrote: post #153
fewhil wrote: post #140

Earlier quoted context omitted.

Isn't Claude Plays Pokemon using image input in addition to text? Not that it's perfect at it (some of its most glaring mistakes are when it just doesn't seem to understand what's on the screen correctly).

Yes but because it's trained on text and in the backend, images are converted to tokens, it is absolutely dogshit at navigation and basic puzzles. It can't figure out what Squirrels can about how to achieve goals in a maze.

The images are converted to an embedding space the size of token embedding space. And the model is trained on that new embedding space. A joint representation of text and images is formed.

It’s not as though the image is converted to text tokens.

Post reply on HN