Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Concept Models: Language modeling in a sentence representation space

github.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Large Concept Models: Language modeling in a sentence representation space

#31

Post by blackeyeblitzar »

mdp2021 wrote: post #12
stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

That should be proven. The two approaches - predicting tokens vs predicting "sentences" - should be compared to see how much their output differ in terms of quality. Edit2: ...and both (and their variants) be compared to other ideas such as "multi-token prediction"... Edit: or, appropriateness of the approach should be demonstrated after acquired "transparency" of how the LLMs effectively internally work. I am not aw…

Isn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?

Re: Large Concept Models: Language modeling in a sentence representation space

#32

Post by mdp2021 »

YeGoblynQueenne wrote: post #26

From the paper: >> In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a “concept”. I wonder if the many authors of the paper know that what they call "concept" is what all of machine learning and AI has also called a "concept" for many decades, and not a new thing that they have just named from scratch. For instance, classes of "co…

I am not sure that fits the point, YGQ:

it seems to me the concept of «concept» in the paper is "the embedding vector we get in systems like SONAR (which we could use to generalize ordered sets of tokens into more complex ideas)". That's pretty specific, only marginally related to past handling as mentioned.

Re: Large Concept Models: Language modeling in a sentence representation space

#33

Post by mdp2021 »

upghost wrote: post #27

Aside from the using the word "concept" instead of "language" I don't see how this is different than an LLM. It's still doing next token prediction. This is like in D&D where you have two swords with wildly different flavor text but ultimately they both do 1d6+1 damage. What am I missing -- aside from the marketing? Is there something architecturally different or what? Looks like regular autoregressive sequence trans…

> something architecturally different

An embedding space engine accepting sentences (SONAR) fit in so that the tokens of this architecture are complex sets of the tokens of past architectures.

Re: Large Concept Models: Language modeling in a sentence representation space

#34

Post by mdp2021 »

blackeyeblitzar wrote: post #31
mdp2021 wrote: post #12

Earlier quoted context omitted.

That should be proven. The two approaches - predicting tokens vs predicting "sentences" - should be compared to see how much their output differ in terms of quality. Edit2: ...and both (and their variants) be compared to other ideas such as "multi-token prediction"... Edit: or, appropriateness of the approach should be demonstrated after acquired "transparency" of how the LLMs effectively internally work. I am not aw…

Isn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?

> multi token prediction of sufficient length

Is multi token prediction the same as predicting the embedding of a complex token (the articulation of those input tokens in a sentence)?

Re: Large Concept Models: Language modeling in a sentence representation space

#35

Post by mdp2021 »

dr_dshiv wrote: post #28
mdp2021 wrote: post #15

Earlier quoted context omitted.

It is explicitly stated in the paper that > One may argue that LLMs are implicitly learning a hierarchical representation, but we stipulate that models with an explicit hierarchical architecture are better suited to create coherent long-form output And the problem remains that (text surrounding the above): > Despite the undeniable success of LLMs and continued progress, all current LLMs miss a crucial characteristic…

I just don’t understand that — I thought deep neural nets were inherently hierarchical. Or at least emergently hierarchical?

Neural Nets can be made to be hierarchical - I would say a most notable example is the Convolutional Neural Network so successfully promoted by Yann Le Cun.

But the issue with the LLMs architectures in place is with the idea of "predicting the next token", so strident with the exercise of intelligence - where we search instead for the "neighbouring fitting ideas".

So, "hierarchical" in this context is there to express that it is typical of natural intelligence to refine an idea - formulating an hypothesis and improving its form (hence its expression) step after step of pondering. The issue of transparency in current LLMs, and the idea of "predicting the next token", do not help in having the idea of typical natural intelligence mechanism and the tentative interpretation of LLM internals match.

Re: Large Concept Models: Language modeling in a sentence representation space

#36

Post by blurbleblurble »

stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

At a performance boost of 10-100x :)

Re: Large Concept Models: Language modeling in a sentence representation space

#37

Post by layer8 »

vimgrinder wrote: post #24

I like the idea of "concept" .. you can represent a concept with language, visual etc. but it isn't any of those. Those are symbols used to communicate a concept or give representation to it but concepts are just connections between other concepts at the core. The closest things i feel to this is categories in category theory.

Concepts need to be linked to reality somehow in order to carry any meaning. They are thus not just relations between themselves.

Re: Large Concept Models: Language modeling in a sentence representation space

#38

Post by YeGoblynQueenne »

mdp2021 wrote: post #32
YeGoblynQueenne wrote: post #26

From the paper: >> In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a “concept”. I wonder if the many authors of the paper know that what they call "concept" is what all of machine learning and AI has also called a "concept" for many decades, and not a new thing that they have just named from scratch. For instance, classes of "co…

I am not sure that fits the point, YGQ: it seems to me the concept of «concept» in the paper is "the embedding vector we get in systems like SONAR (which we could use to generalize ordered sets of tokens into more complex ideas)". That's pretty specific, only marginally related to past handling as mentioned.

That's only the representation of a concept. Different systems and different approaches will have different representations but that doesn't change the fact of what is being represented.

Re: Large Concept Models: Language modeling in a sentence representation space

#40

Post by nightski »

mdp2021 wrote: post #35
dr_dshiv wrote: post #28

Earlier quoted context omitted.

I just don’t understand that — I thought deep neural nets were inherently hierarchical. Or at least emergently hierarchical?

Neural Nets can be made to be hierarchical - I would say a most notable example is the Convolutional Neural Network so successfully promoted by Yann Le Cun. But the issue with the LLMs architectures in place is with the idea of "predicting the next token", so strident with the exercise of intelligence - where we search instead for the "neighbouring fitting ideas". So, "hierarchical" in this context is there to expres…

Is that true? There are many attention/mlp layers stacked on top of each other. Higher level layers aren't performing attention on input tokens, but instead on the output of the previous layer.
Post reply on HN