Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From word models to world models

arxiv.org

Post reply on HN
Print view View original

111–119 of 119 posts

Re: From word models to world models

#111

Post by YeGoblynQueenne »

gibsonf1 wrote: post #110
YeGoblynQueenne wrote: post #97

Earlier quoted context omitted.

In general, in AI, when we talk about "concepts" we're talking about the things machine learning models are trained to, well, to model. In PAC-Learning terms, specifically, a "concept" is a set of instances (which may be vectors or whatever). Note that a "concept" is not the same as a "class", as in classification. Instead a concept belongs to a class of similar concepts and a learner is trained on instances of conce…

I think modeling a human concept would be a far better approach.

That is indeed the general idea. That's why they're called "concepts", they're meant to be things, or categories of things, that we perceive. But "concept" also has a technical sense, of the assumed representation in machine learning.

That is, in machine learning a concept is represented as a set of instances. Inside the human mind, who knows.

Re: From word models to world models

#112

Post by jbay808 »

YeGoblynQueenne wrote: post #107
jbay808 wrote: post #102

Earlier quoted context omitted.

> what do you think would happen if you gave your model an alphanumeric list to sort? Did you try that? The tokenizer would throw an exception, because it doesn't have any tokens to represent alphabetical characters. But you tell me - if I had tokenized alphabetical characters and defined an ordering, would you expect the results to be any different? > You say e.g. that "LLM is learning an n-gram"[...] you can't "lea…

>> Where do I say that? In your comment above: (...) is expressed a little bit more clearly as _the LLM is learning an n-gram_ that produces correct sorts (...) (My underlining) You also use it in a similarly unusual way throughout your linked substack post, for example, you write: the way GPT works is, in a certain sense, functionally equivalent to an n-gram, but that doesn’t mean GPT is an n-gram. Where does this u…

> In your comment above

Oh, I see, good catch. I think that comment was a result of a botched edit; I do that sometimes. Too late to change it now. Sorry for the confusion!

> Where does this use of "n-gram" come from? I mean, did you see it somewhere?

It's shorthand for n-gram Markov model. The same way it is presented in, for example, A Mathematical Theory of Communication.

> "Defined an ordering", where?

In order for a set to be sortable, you need to define an ordering over the elements. So for example, defining that the letter 'A' is greater than the number '99'. It's easy to take for granted that 1 a priori, because the tokens are just index values. It doesn't have any way to know that token number 5 represents the character '5'.

> if you take your model that's trained on two-digit lists of integers and you run it on lists of any other type of elements it will not be able to sort them correctly.

To reiterate, the token dictionary basically just contains the characters "0123456789,():[]_\n". If you try to ask it to sort '(Tuesday, Monday)', it's just going to throw an exception because 'T' isn't a recognized token; it doesn't have a corresponding index. It's not even a question of whether it can sort them correctly or incorrectly.

> "Any input list"? How so?

I think the meaning is pretty clear. No algorithm can sort a list of elements that aren't members of a totally ordered set, so I wasn't attempting to imply that any input list meant that a neural network could somehow supersede this limitation.

Re: From word models to world models

#113

Post by jbay808 »

mjburgess wrote: post #105
jbay808 wrote: post #104

Earlier quoted context omitted.

> The (sequential) distribution of digits amongst sorted numbers is tiny This is why 10^80 random lists gets reduced to only 10^36 sorted lists. However, 10^36 is still very large with respect to the size of the model.

You're treating each list as unique, all the lists have a distribution of digits in common... I'm at a loss to even understand what you're saying here really -- this is why you need to actually state, formally, what you think the "LLMs are just stats" hypothesis amounts to. It seems you think it amounts to saying LLMs sample from a combinatorial space, naively construed -- but that isnt the claim? The claim is rather…

I agree a neural network can certainly learn the conditional distributions that let it make that choice correctly every time. Once it has done so, then do you not have a sorting algorithm?

Re: From word models to world models

#114

Post by jbay808 »

YeGoblynQueenne wrote: post #108
jbay808 wrote: post #101

Earlier quoted context omitted.

> it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). > Also, Machine Learning 101: you test your models on a test set that is disjoint to…

>> Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). That's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim. What is certainly established is that a function, and an algorithm, are different objects. To clarify, a f…

> that's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim.

Would you change your mind for a different link, like this one? http://proceedings.mlr.press/v139/weiss21a.html

I think you would enjoy learning about RASP, rather than taking such a hardline skeptical position.

> a function is a mapping between the elements of two sets, whereas an algorithm is a sequence of operations that calculates the result of a function and is guaranteed to terminate

I'm aware. Transformers (and RASP programs) are guaranteed to terminate; that's one of their nice properties.

> Is there a good reason not to do that?

Balanced against the value of my unpaid time, a probability of 10^-70 is low enough for the purposes of a quick and fun test.

Speaking of which, I'm going to enjoy my weekend now. I hope you enjoy yours!

Re: From word models to world models

#115

Post by dimatura »

moffkalast wrote: post #80
dimatura wrote: post #66

Earlier quoted context omitted.

I disagree that this is an "unfounded hot take". It's far from a rare opinion on cognitive science, and if I had to guess it's probably the mainstream opinion (I can't really back that up with citations because I haven't followed the field closely in the last decade). And for what it's worth, I agree with Yann, although I have to admit that LLMs work far better than I would've guessed. It's a topic that's too large f…

Well I admit I used to be of a similar opinion as well, but seeing this explosion unravel over the past few months has me convinced that it's can't possibly be right, at least not to any degree that objectively matters. Perhaps language is the wrong term to use, since it's not what LLMs are really about. They're about text. There are very few things that cannot be expressed as text, albeit in unconventional ways like…

Well, that idea is one of the motivations behind the paper, which is itself a throwback to earlier ideas about the "language of thought", a hypothetical language (certainly not ordinary natural language, and probably more like a programming language). But adding a few twists such as the probabilistic part and of course the whole machinery of LLMs, and more emphasis on sensory grounding. I think it's a very interesting approach from a researcher I respect, but obviously don't know if it'll pan out.

Re: From word models to world models

#116

Post by Al0neStar »

jbay808 wrote: post #114
YeGoblynQueenne wrote: post #108

Earlier quoted context omitted.

>> Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). That's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim. What is certainly established is that a function, and an algorithm, are different objects. To clarify, a f…

> that's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim. Would you change your mind for a different link, like this one? http://proceedings.mlr.press/v139/weiss21a.html I think you would enjoy learning about RASP, rather than taking such a hardline skeptical position. > a function is a mapp…

That's the same paper.

Re: From word models to world models

#117

Post by YeGoblynQueenne »

Al0neStar wrote: post #116
jbay808 wrote: post #114

Earlier quoted context omitted.

> that's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim. Would you change your mind for a different link, like this one? http://proceedings.mlr.press/v139/weiss21a.html I think you would enjoy learning about RASP, rather than taking such a hardline skeptical position. > a function is a mapp…

That's the same paper.

EDIT: I realise I was mistaken about the OP. He is not an undergarduate student, as I initially thought. His substack profile says he is a professional engineer and consultant. So his complete cluelessness about computer science fundamentals is not the result of inexperience, and his article is nothing more than an attempt to jump on the current bandwagon of LLM hype rather than an attempt to make sense of things. I thought I was helping a CS grad learn something! What an idiot I am! Fuck. φτου γαμώ την Παναγία μου.

[Earlier text of my comment left in the interest of something or other]

It is, but it's published in the proceedings of the ICML, which means it's been peer reviewed.

The OP has checked out (I guess all this computer scienc-y stuff is boring on a weekend), but even a peer-reviewed article is not enough to cause us to let go of good, old-fashioned computer science. The article basically invents its own language and then proceeds to map transformers to it, to claim that transformers can learn various kinds of programs. It's not convincing.

In any case, learning to sort lists by neural nets is not something new, or unique to transformers, and there's pretty clear understanding of how it works. I explain why it doesn't constitute learning an algorithm in my comment above. The RASP paper doesn't change that. I mean, Recurrent Neural Nets have a known equivalence to FSMs but even they cannot learn algorithms but only approximate them. The OP wrote his article in an obvious effort to understand why GPT is "not an n-gram" even if it behaves like an n-gram model (well, it's a language model, it doesn't matter what it's trained on) so I'm guessing he can appreciate the need for clarity in explaining empirical results and he probably will want to think further on what, exactly, his experiment has shown. I hope my little comment above will help him do that.

Re: From word models to world models

#118

Post by mjburgess »

jbay808 wrote: post #113
mjburgess wrote: post #105

Earlier quoted context omitted.

You're treating each list as unique, all the lists have a distribution of digits in common... I'm at a loss to even understand what you're saying here really -- this is why you need to actually state, formally, what you think the "LLMs are just stats" hypothesis amounts to. It seems you think it amounts to saying LLMs sample from a combinatorial space, naively construed -- but that isnt the claim? The claim is rather…

I agree a neural network can certainly learn the conditional distributions that let it make that choice correctly every time. Once it has done so, then do you not have a sorting algorithm?

So this is what I thought you would say, and it's the origin of the issues here: to say that LLMs are "statistical parrots" is just to say they learn conditional distributions of text tokens.

So you aren't replying to the "only stats" claim: that is the claim!

The issue is that language-use isn't a matter of distributions of text tokens: when i say, "the sky is clear today!" it is caused by there being a blue sky. Then I say, "therefore I'd like to go out!" it is caused by my preferences, etc.

So if we had a generative causal model of language it would be something like this: Agent + Environment + Representations ---SymbolicTranslation---> Language.

All LLMs do is model the data being generated by this process, they dont model the process (ie., agents, environments, representations, etc.)

They say, "it is a nice day" only because those tokens match some statistical distribution over historical texts. Not because it has judged the day nice.

To model language is not to provide an indistinguishable language-like distribution of text tokens, but rather, for an agent to use language to express ideas caused by their internal states + the world.

In the case of sorting numbers, the tokens themselves have the property (ie., mathematical properties such as ranking are had by ranked tokens). So learning the distribution is learning the property of interest.

This is why no papers which demonstrate NNs "have representations" etc. which appeal to formal properties the data itself has, are even releveant to the discussion. Yet, all this "world model, algorithm, blah blah" said of NNs, is only ever shown using data whose "unsupervised model" constitues the property of interest.

Statistical models of the distributions of tokens are not models of the data generating process which produces those tokens (unless that process is just the distribution of those tokens). This is obvious from the outset.

Re: From word models to world models

#119

Post by gibsonf1 »

canjobear wrote: post #91
gibsonf1 wrote: post #89

Earlier quoted context omitted.

Right, derived from word pattern statistics. The CYC project tried first order predicate calculus with complete failure. This is not how we think or how conceptual awareness works. The key give away is what they don't talk about, Concepts.

They talk about concepts several times, defined as probabilistic functions within their LOT.

Except human concepts are not probablistic functions.
Post reply on HN