What used to be feature engineering a decade or more ago now seems to have shifted to developing distributed representations. LLMs use word tokens (for words or the entities in images). But there are many more. The 3D Fields (or whatever they have evolved to) developed by Fei-Fei Li's group represent visual information in a way better suited for geometrical tasks. Wav2Vec, the convolutional features for YOLO and friends, and these sentence representations are other examples. I would love to read a review of this circle of ideas.
Large Concept Models: Language modeling in a sentence representation space
51–60 of 61 posts
Re: Large Concept Models: Language modeling in a sentence representation space
#52Earlier quoted context omitted.
Everyone keeps claiming this but we have zero evidence of any kind of scaling wall what-so-ever. Oh you mean data? Synthetic Data, Agents, and Digitization solve that.
There were multiple reports confirming that OpenAI's Orion (planned to be GPT-5) yielded unexpectedly weak results.
And costs $500 million per training run.
Re: Large Concept Models: Language modeling in a sentence representation space
#53Earlier quoted context omitted.
Everyone keeps claiming this but we have zero evidence of any kind of scaling wall what-so-ever. Oh you mean data? Synthetic Data, Agents, and Digitization solve that.
There were multiple reports confirming that OpenAI's Orion (planned to be GPT-5) yielded unexpectedly weak results.
So Deepseek V3 did nothing to show you how wrong this take is?
Re: Large Concept Models: Language modeling in a sentence representation space
#54This maps a little to what we are doing research on what we are calling as shape of stories[1]. We can clearly see in 2D space itself how different "concepts" are explored. Using the shape of stories for semantic chunking we can clearly see in multiple articles how we can chunk by "concepts". [2] Now we are trying to see if we can just use these chunks and train a next "chunk" predictor instead of a next word predict…
Can you spot conceptually similar stories by their shape?
For instance what is the shape of the ugly duckling compared to Rudolf the red nosed reindeer. They are essentially the same story, so presumably on some dimension you should be able to spot them in a group of unrelated stories.
Re: Large Concept Models: Language modeling in a sentence representation space
#55Aside from the using the word "concept" instead of "language" I don't see how this is different than an LLM. It's still doing next token prediction. This is like in D&D where you have two swords with wildly different flavor text but ultimately they both do 1d6+1 damage. What am I missing -- aside from the marketing? Is there something architecturally different or what? Looks like regular autoregressive sequence trans…
(Guessing here) It does tokenization and prediction for a whole sentence, not fragments of words. I like this idea because that's how humans think. We mentally formulate a whole sentence, then say it. People who don't do this speak in run-ons and word salad.
I would be interested to know how many people do formulate a whole sentence before saying it. "Think before you speak" as they say. I feel I do not have the cognitive window or processing speed to do this; instead, I formulate a concept of how I would like to respond abstractly, and then think of and say phrases of several words one at a time until the sentence ends itself. The latter process leans heavily on some kind of next word anticipation.
Re: Large Concept Models: Language modeling in a sentence representation space
#56I like the idea of "concept" .. you can represent a concept with language, visual etc. but it isn't any of those. Those are symbols used to communicate a concept or give representation to it but concepts are just connections between other concepts at the core. The closest things i feel to this is categories in category theory.
Platonic forms?
interesting concept they are.
Re: Large Concept Models: Language modeling in a sentence representation space
#57This maps a little to what we are doing research on what we are calling as shape of stories[1]. We can clearly see in 2D space itself how different "concepts" are explored. Using the shape of stories for semantic chunking we can clearly see in multiple articles how we can chunk by "concepts". [2] Now we are trying to see if we can just use these chunks and train a next "chunk" predictor instead of a next word predict…
Can you spot conceptually similar stories by their shape? For instance what is the shape of the ugly duckling compared to Rudolf the red nosed reindeer. They are essentially the same story, so presumably on some dimension you should be able to spot them in a group of unrelated stories.
Will check for these particular stories. But yes, when we tried this on some stories with a similar arc we saw that their path is similar in the semantic space.
Re: Large Concept Models: Language modeling in a sentence representation space
#58Earlier quoted context omitted.
It is explicitly stated in the paper that > One may argue that LLMs are implicitly learning a hierarchical representation, but we stipulate that models with an explicit hierarchical architecture are better suited to create coherent long-form output And the problem remains that (text surrounding the above): > Despite the undeniable success of LLMs and continued progress, all current LLMs miss a crucial characteristic…
I just don’t understand that — I thought deep neural nets were inherently hierarchical. Or at least emergently hierarchical?
Not necessarily.
For visual tasks, that is the state of the art, with visual features being "gouped" into more semantically relevant parts ("circles" grouped into "fluffy textures" grouped into "dog ears"). This hierarchy building behavior is baked into the model.
For transformers, not so much. Although each transformer block output serve as input for the next block, they can learn hierarchical relationship (in latent space, not in human language), but that is not backed nor enforced in the architecture.
Re: Large Concept Models: Language modeling in a sentence representation space
#59Earlier quoted context omitted.
(Guessing here) It does tokenization and prediction for a whole sentence, not fragments of words. I like this idea because that's how humans think. We mentally formulate a whole sentence, then say it. People who don't do this speak in run-ons and word salad.
I would be interested to know how many people do formulate a whole sentence before saying it. "Think before you speak" as they say. I feel I do not have the cognitive window or processing speed to do this; instead, I formulate a concept of how I would like to respond abstractly, and then think of and say phrases of several words one at a time until the sentence ends itself. The latter process leans heavily on some ki…
> how many people do formulate a whole sentence before saying it
The process is a formulation of precise ideas (complex at some level and verified to some degree, hopefully), then translated into sentences for output (not necessarily in these two steps, but through iterations).
This project tries to use sentences as formalizations of ideas - an interesting way enabled by availability of tools, allowing good features like transparency.
Re: Large Concept Models: Language modeling in a sentence representation space
#60Earlier quoted context omitted.
50B still seems reasonable compared to the revenue of the Big AI companies.
what revenues? If by big AI companies you mean llm service providers (OpenAI, ...), their revenues are far from high or profitable. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t... Maybe Nvidia, but they are a chip / hardware maker first. And even for them 50B training run with no exponential gains seems unreasonable. Better to optimize the architecture / approach first, which also is what most compani…
It's not unusual to make infrastructure investments that will pay off in 30-50 years. I don't see why not an AI model - unless it's not true that we're at the end of scaling.