Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Concept Models: Language modeling in a sentence representation space

github.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Large Concept Models: Language modeling in a sentence representation space

#21

Post by mdp2021 »

Jensson wrote: post #20
stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

> You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced. That is what tokens are doing in the first place though, and you get better results with tokens instead of letters.

Well, individual letters in these languages in use* do not convey specific meaning, while individual tokens do - so, you cannot really construe a ladder that would go from letter to token, then from token to sentence.

This said, to research whether the search for concepts (in the solutions space) works better than the search for tokens seems absolutely dutiful, in absence of a solid theory that showed otherwise.

(*Sounds convey their own meaning e.g. in proto-Indo-European according to some interpretations, but that becomes too remote in the current descendants - you cannot reconstruct the implicit sound-token in words directly in English, just from the spelling.)

Re: Large Concept Models: Language modeling in a sentence representation space

#22

Post by IanCal »

Jensson wrote: post #20
stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

> You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced. That is what tokens are doing in the first place though, and you get better results with tokens instead of letters.

Is that true? I thought there was a desire to move towards byte level work rather than tokens, and that the benefits of tokens was more that you are reducing the context size for the same input.

Re: Large Concept Models: Language modeling in a sentence representation space

#24

Post by vimgrinder »

I like the idea of "concept" .. you can represent a concept with language, visual etc. but it isn't any of those. Those are symbols used to communicate a concept or give representation to it but concepts are just connections between other concepts at the core. The closest things i feel to this is categories in category theory.

Re: Large Concept Models: Language modeling in a sentence representation space

#25

Post by motoboi »

mdp2021 wrote: post #15
stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

It is explicitly stated in the paper that > One may argue that LLMs are implicitly learning a hierarchical representation, but we stipulate that models with an explicit hierarchical architecture are better suited to create coherent long-form output And the problem remains that (text surrounding the above): > Despite the undeniable success of LLMs and continued progress, all current LLMs miss a crucial characteristic…

I suppose humans need high level concepts because we can only hold 7[] things in working memory. Computers don’t have that limitation.

Also, humans cannot iterate over thousands of possibilities in a second, like computers do.

And finally, animal brains are severely limited by heat dissipation and energy input flow.

Based on that, artificial intelligence may arise from unexpected simple strategies, given the fundamental differences in scale and structure from animal brains.

- where 7 is whatever number is the correct number nowadays.

Re: Large Concept Models: Language modeling in a sentence representation space

#26

Post by YeGoblynQueenne »

From the paper:

>> In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a “concept”.

I wonder if the many authors of the paper know that what they call "concept" is what all of machine learning and AI has also called a "concept" for many decades, and not a new thing that they have just named from scratch.

For instance, classes of "concepts" are the target of learning in Leslie Valiant's "A Theory of the Learnable", the paper that introduced Probably Approximately Correct Learning (PAC-Learning). Quoting from its abstract:

  ABSTRACT: Humans appear to be able to learn new
  concepts without needing to be programmed explicitly in
  any conventional sense. In this paper we regard learning as
  the phenomenon of knowledge acquisition in the absence of
  explicit programming. We give a precise methodology for
  studying this phenomenon from a computational viewpoint.
  It consists of choosing an appropriate information gathering
  mechanism, the learning protocol, and exploring the class of
  concepts that can be learned using it in a reasonable
  (polynomial) number of steps. Although inherent algorithmic
  complexity appears to set serious limits to the range of
  concepts that can be learned, we show that there are some
  important nontrivial classes of propositional concepts that
  can be learned in a realistic sense
From: https://web.mit.edu/6.435/www/Valiant84.pdf

Or take this Introduction to Chapter 2 in Tom Mitchell's "Machine Learning" (the original ML textbook, published 1997):

  This chapter considers concept learning: acquiring the definition of 
  a general category given a sample of positive and negative training 
  examples of the category.
From: https://www.cs.cmu.edu/~tom/mlbook.html (clink link in "the book").

I mean I really wonder some times what is going on here. There's been decades of research in AI and machine learning but recently papers look like their authors have landed in an undiscovered country and are having to invent everything from scratch. That's not good. There are pitfalls that all the previous generations have explored thoroughly by falling in them time and again. Those who don't remember those lessons will have to find that out the hard way.

Re: Large Concept Models: Language modeling in a sentence representation space

#27

Post by upghost »

Aside from the using the word "concept" instead of "language" I don't see how this is different than an LLM. It's still doing next token prediction. This is like in D&D where you have two swords with wildly different flavor text but ultimately they both do 1d6+1 damage.

What am I missing -- aside from the marketing? Is there something architecturally different or what? Looks like regular autoregressive sequence transformer to me.

Re: Large Concept Models: Language modeling in a sentence representation space

#28

Post by dr_dshiv »

mdp2021 wrote: post #15
stravant wrote: post #11

This feels like a failure to learn the bitter lesson: You're just taking the translation to concepts that the LLM is certainly already doing and trying to make it explicitly forced.

It is explicitly stated in the paper that > One may argue that LLMs are implicitly learning a hierarchical representation, but we stipulate that models with an explicit hierarchical architecture are better suited to create coherent long-form output And the problem remains that (text surrounding the above): > Despite the undeniable success of LLMs and continued progress, all current LLMs miss a crucial characteristic…

I just don’t understand that — I thought deep neural nets were inherently hierarchical. Or at least emergently hierarchical?

Re: Large Concept Models: Language modeling in a sentence representation space

#29

Post by tantalor »

upghost wrote: post #27

Aside from the using the word "concept" instead of "language" I don't see how this is different than an LLM. It's still doing next token prediction. This is like in D&D where you have two swords with wildly different flavor text but ultimately they both do 1d6+1 damage. What am I missing -- aside from the marketing? Is there something architecturally different or what? Looks like regular autoregressive sequence trans…

(Guessing here) It does tokenization and prediction for a whole sentence, not fragments of words.

I like this idea because that's how humans think. We mentally formulate a whole sentence, then say it. People who don't do this speak in run-ons and word salad.

Re: Large Concept Models: Language modeling in a sentence representation space

#30

Post by upghost »

tantalor wrote: post #29
upghost wrote: post #27

Aside from the using the word "concept" instead of "language" I don't see how this is different than an LLM. It's still doing next token prediction. This is like in D&D where you have two swords with wildly different flavor text but ultimately they both do 1d6+1 damage. What am I missing -- aside from the marketing? Is there something architecturally different or what? Looks like regular autoregressive sequence trans…

(Guessing here) It does tokenization and prediction for a whole sentence, not fragments of words. I like this idea because that's how humans think. We mentally formulate a whole sentence, then say it. People who don't do this speak in run-ons and word salad.

oh interesting. concepts as tokens. Yeah I'd buy that. They do something similar with transformers in robotics, except they use tokens as actions instead of word chunks. Good eye.
Post reply on HN