Large Concept Models: Language modeling in a sentence representation space
1–10 of 61 posts
Re: Large Concept Models: Language modeling in a sentence representation space
#2This is interesting. I wonder if such a project could dive into lower-level concepts, those akin to prime numbers. The atoms from which all other concepts are built.
Re: Large Concept Models: Language modeling in a sentence representation space
#3Between this and learned patches and ModernBERT and DeepSeek?
I think it’s time to read up.
Re: Large Concept Models: Language modeling in a sentence representation space
#4This is like going back to CNNs. Attention is all you need.
Re: Large Concept Models: Language modeling in a sentence representation space
#5This is like going back to CNNs. Attention is all you need.
Quantum states are all one really needs, but it turns out that it's way to computationally expensive to simulate all that just for the purpose of AI applications - so instead we have to go to higher levels of construction. Attention is surely just about on the cusp of what is computationally reasonable which means that it's not all we need, we need more efficient and richer constructions.
Re: Large Concept Models: Language modeling in a sentence representation space
#6This is like going back to CNNs. Attention is all you need.
Quantum states are all one really needs, but it turns out that it's way to computationally expensive to simulate all that just for the purpose of AI applications - so instead we have to go to higher levels of construction. Attention is surely just about on the cusp of what is computationally reasonable which means that it's not all we need, we need more efficient and richer constructions.
Yes, just spray Quantum on it
Re: Large Concept Models: Language modeling in a sentence representation space
#7This is like going back to CNNs. Attention is all you need.
Attention is just communication? It’s orthogonal to the space of the representation.
Re: Large Concept Models: Language modeling in a sentence representation space
#8Earlier quoted context omitted.
Quantum states are all one really needs, but it turns out that it's way to computationally expensive to simulate all that just for the purpose of AI applications - so instead we have to go to higher levels of construction. Attention is surely just about on the cusp of what is computationally reasonable which means that it's not all we need, we need more efficient and richer constructions.
Yes, just spray Quantum on it
> Yes, just spray Quantum on it
Careful, don’t give Sam Altman any ideas.
Once OpenAI cannot raise enough capital, he will aim quantum AGI.
Re: Large Concept Models: Language modeling in a sentence representation space
#9> Current best practice for large scale language modeling is to operate at the token level, i.e. to learn to predict the next tokens given a sequence of preceding tokens. There is a large body of research on improvements of LLMs, but most works concentrate on incremental changes and do not question the main underlying architecture. In this paper, we have proposed a new architecture,
For some 2024 may have ended badly,
but reading the lines above shines a great light of hope for the new year.
Re: Large Concept Models: Language modeling in a sentence representation space
#10This is like going back to CNNs. Attention is all you need.
Quantum states are all one really needs, but it turns out that it's way to computationally expensive to simulate all that just for the purpose of AI applications - so instead we have to go to higher levels of construction. Attention is surely just about on the cusp of what is computationally reasonable which means that it's not all we need, we need more efficient and richer constructions.
We do not need quantum states to build (arithmetic) calculators. Nor, very probably, for complex and much more complex calculators.