Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Concept Models: Language modeling in a sentence representation space

github.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Large Concept Models: Language modeling in a sentence representation space

#41

Post by mdp2021 »

nightski wrote: post #40
mdp2021 wrote: post #35

Earlier quoted context omitted.

Neural Nets can be made to be hierarchical - I would say a most notable example is the Convolutional Neural Network so successfully promoted by Yann Le Cun. But the issue with the LLMs architectures in place is with the idea of "predicting the next token", so strident with the exercise of intelligence - where we search instead for the "neighbouring fitting ideas". So, "hierarchical" in this context is there to expres…

Is that true? There are many attention/mlp layers stacked on top of each other. Higher level layers aren't performing attention on input tokens, but instead on the output of the previous layer.

> Is that true

Well, if you are referring to «The issue of transparency in current LLMs», I have not read an essay that explains satisfactorily the inner process and world modelling inside LLMs. Some pieces say (guess?) that the engine has no idea what the whole concept in the reply would be before outputting all the tokens, others swear it seems impossible it has no such idea before formulation...

Re: Large Concept Models: Language modeling in a sentence representation space

#42

Post by nutanc »

This maps a little to what we are doing research on what we are calling as shape of stories[1].

We can clearly see in 2D space itself how different "concepts" are explored.

Using the shape of stories for semantic chunking we can clearly see in multiple articles how we can chunk by "concepts". [2]

Now we are trying to see if we can just use these chunks and train a next "chunk" predictor instead of a next word predictor.

In the paper, they take a sentence to mean a concept. We believe that a "semantic chunk" is better suited for a concept instead of a sentence.

[1] https://gpt3experiments.substack.com/p/the-shape-of-stories-...

[2]https://gpt3experiments.substack.com/p/a-new-chunking-approa...

Re: Large Concept Models: Language modeling in a sentence representation space

#43

Post by mdp2021 »

YeGoblynQueenne wrote: post #38
mdp2021 wrote: post #32

Earlier quoted context omitted.

I am not sure that fits the point, YGQ: it seems to me the concept of «concept» in the paper is "the embedding vector we get in systems like SONAR (which we could use to generalize ordered sets of tokens into more complex ideas)". That's pretty specific, only marginally related to past handling as mentioned.

That's only the representation of a concept. Different systems and different approaches will have different representations but that doesn't change the fact of what is being represented.

But if the issue is about "research in AI has had to deal with the concept of "concept" since the inception" (and of course it had to), the contribution in this paper is to try an operational implementation that could bear fruit and possibly fix architectural shortcomings of the mainstream effort.

(It is not separate from the context of LLMs.)

Re: Large Concept Models: Language modeling in a sentence representation space

#44

Post by fngjdflmdflg »

IanCal wrote: post #22
Jensson wrote: post #20

Earlier quoted context omitted.

> You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced. That is what tokens are doing in the first place though, and you get better results with tokens instead of letters.

Is that true? I thought there was a desire to move towards byte level work rather than tokens, and that the benefits of tokens was more that you are reducing the context size for the same input.

>there was a desire to move towards byte level work rather than tokens

Yeah, latest work on this is from Meta a last month.[0] It showed good results.

[0] https://ai.meta.com/research/publications/byte-latent-transf... (https://news.ycombinator.com/item?id=42415122)

Re: Large Concept Models: Language modeling in a sentence representation space

#45

Post by throwawaymaths »

mdp2021 wrote: post #35
dr_dshiv wrote: post #28

Earlier quoted context omitted.

I just don’t understand that — I thought deep neural nets were inherently hierarchical. Or at least emergently hierarchical?

Neural Nets can be made to be hierarchical - I would say a most notable example is the Convolutional Neural Network so successfully promoted by Yann Le Cun. But the issue with the LLMs architectures in place is with the idea of "predicting the next token", so strident with the exercise of intelligence - where we search instead for the "neighbouring fitting ideas". So, "hierarchical" in this context is there to expres…

there is a way that "predicting the next token" is ~append-only turing machine. Obviously the tokens we're using might be suboptimal for whatever goalpost "agi" is at any given time, but the structure/strategies of LLMs is probably not far from a really good one, modulo refactoring for efficiency like MAMBA (but still doing token stream prediction, esp. during inference)

Re: Large Concept Models: Language modeling in a sentence representation space

#46

Post by blackeyeblitzar »

mdp2021 wrote: post #34
blackeyeblitzar wrote: post #31

Earlier quoted context omitted.

Isn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?

> multi token prediction of sufficient length Is multi token prediction the same as predicting the embedding of a complex token (the articulation of those input tokens in a sentence)?

To be honest I don’t know. Maybe the only way to know is to build and measure all these variations.

Re: Large Concept Models: Language modeling in a sentence representation space

#47

Post by YeGoblynQueenne »

mdp2021 wrote: post #43
YeGoblynQueenne wrote: post #38

Earlier quoted context omitted.

That's only the representation of a concept. Different systems and different approaches will have different representations but that doesn't change the fact of what is being represented.

But if the issue is about "research in AI has had to deal with the concept of "concept" since the inception" (and of course it had to), the contribution in this paper is to try an operational implementation that could bear fruit and possibly fix architectural shortcomings of the mainstream effort. (It is not separate from the context of LLMs.)

Right, but there's been many operationalisations before. That's what's not new. Tome Mitchell's textbook has plenty of examples. Basically all of machine learning is about learning concepts- in practice as well as in theory. That's the whole point.

Re: Large Concept Models: Language modeling in a sentence representation space

#48

Post by pegasus »

cubefox wrote: post #17
Der_Einzige wrote: post #14

Earlier quoted context omitted.

Everyone keeps claiming this but we have zero evidence of any kind of scaling wall what-so-ever. Oh you mean data? Synthetic Data, Agents, and Digitization solve that.

There were multiple reports confirming that OpenAI's Orion (planned to be GPT-5) yielded unexpectedly weak results.

And not just OpenAI is facing this problem. Anthropic and Google as well.

Re: Large Concept Models: Language modeling in a sentence representation space

#49

Post by dr_dshiv »

vimgrinder wrote: post #24

I like the idea of "concept" .. you can represent a concept with language, visual etc. but it isn't any of those. Those are symbols used to communicate a concept or give representation to it but concepts are just connections between other concepts at the core. The closest things i feel to this is categories in category theory.

Platonic forms?

Re: Large Concept Models: Language modeling in a sentence representation space

#50

Post by UltraSane »

Der_Einzige wrote: post #14
anon373839 wrote: post #13

Earlier quoted context omitted.

The bitter lesson isn’t a law of nature, though. And as GPT-style LLMs appear to be at the foot of a scaling wall, I personally think inductive bias is due for a comeback.

Everyone keeps claiming this but we have zero evidence of any kind of scaling wall what-so-ever. Oh you mean data? Synthetic Data, Agents, and Digitization solve that.

There seems to be a affordable scaling wall.
Post reply on HN