Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From word models to world models

arxiv.org

Post reply on HN
Print view View original

101–110 of 119 posts

Re: From word models to world models

#101

Post by jbay808 »

YeGoblynQueenne wrote: post #94
jbay808 wrote: post #25

Earlier quoted context omitted.

> It is absolutely trivial to show Hyp2 is false To investigate precisely this question in a clear and unambiguous way, I trained an LLM from scratch to sort lists of numbers. It learned to sort them correctly, and the entropy is such that it's absolutely impossible that it could have done this by Hyp1 (sampling from similar text in the training set). https://jbconsulting.substack.com/p/its-not-just-statistics-... No…

Your model is not sorting correctly and it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules…

> it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners.

Transformers are RASP programs, which includes sorting programs. See the Weiss paper (https://arxiv.org/pdf/2106.06981.pdf).

> Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules, but because, by testing the model on held-out data, we can predict the error the model will have on unseen data

The probability of a test list existing in the training set is less than 10^-70.

Re: From word models to world models

#102

Post by jbay808 »

YeGoblynQueenne wrote: post #95
jbay808 wrote: post #37

Earlier quoted context omitted.

> you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something" Let's not be so hasty. I think I do put it as clearly as possible. I'm comparing essentially your Hyp1 and Hyp2, where Hyp1 (aka the stochastic parrot ) is expressed a little bit more clearly as the LLM is learning an n -gram that produces correct sorts through rote memorization of statistical correlati…

I'm curious, why are you using "n-gram" as if you're referring to a model? You say e.g. that "LLM is learning an n-gram". N-grams are features, not models. You can train an n-gram model, or you can train a language model using n-grams as features, and so on, but you can't "learn an n-gram". Where did you find this terminology? EDIT: >> and Hyp2 is that the LLM's training molds it into representing an actual sorting a…

> what do you think would happen if you gave your model an alphanumeric list to sort? Did you try that?

The tokenizer would throw an exception, because it doesn't have any tokens to represent alphabetical characters. But you tell me - if I had tokenized alphabetical characters and defined an ordering, would you expect the results to be any different?

> You say e.g. that "LLM is learning an n-gram"[...] you can't "learn an n-gram".

Where do I say that? I don't think I make any reference to "learning an n-gram", which is a relief because I don't know what it would mean to "learn an n-gram".

> There's plenty of error in the figure where you show its accuracy (not clear if that's training or test accuracy).

Test accuracy between training iterations (not part of the training process itself, which uses its own separate validation set which is split from the training set). And yes, I agree, it is not error-free, and I wouldn't expect it to be, especially after so little training. What the figure shows is the percentage of sorts that were error-free, and how rapidly that decreases. I've since repeated the test with finer resolution, and the fraction of imperfect sorts continues to decrease about as you expect, which is enough to satisfy my curiosity, although I'm a little curious to see if there is some point where it falls completely to zero.

Re: From word models to world models

#103

Post by [deleted] »

mjburgess wrote: post #100
jbay808 wrote: post #68

Earlier quoted context omitted.

> If LLMs "learned algorithms", the best compression would be on the order of bytes. Yes. Except: (1) the model size is fixed during training, it would be impossible to obtain a bytes-sized result regardless of what it learns to represent. One might even open the thing up and find bubblesort* inside followed by 599 MB of junk DNA; that size is dictated by how it was initialized. (2) I'm not claiming this model is a m…

> Can you provide more details about what you mean by this distributional structure The distribution of sorted digits is: (0 1 2 3 4 5 6 7 8 9) before (1 before 0 1 2 3 4 5 6 7 8 9) before (2 before 0 1 2 3 4 5 6 7 8 9) before (3 before 0 1 2 3 4 5 6 7 8 9) ... ... When you compute the search space you're treating each number as a unique token (ie., that all ordinals are unique) -- but its not sorting unique ordinals…

[deleted]

Re: From word models to world models

#104

Post by jbay808 »

mjburgess wrote: post #100
jbay808 wrote: post #68

Earlier quoted context omitted.

> If LLMs "learned algorithms", the best compression would be on the order of bytes. Yes. Except: (1) the model size is fixed during training, it would be impossible to obtain a bytes-sized result regardless of what it learns to represent. One might even open the thing up and find bubblesort* inside followed by 599 MB of junk DNA; that size is dictated by how it was initialized. (2) I'm not claiming this model is a m…

> Can you provide more details about what you mean by this distributional structure The distribution of sorted digits is: (0 1 2 3 4 5 6 7 8 9) before (1 before 0 1 2 3 4 5 6 7 8 9) before (2 before 0 1 2 3 4 5 6 7 8 9) before (3 before 0 1 2 3 4 5 6 7 8 9) ... ... When you compute the search space you're treating each number as a unique token (ie., that all ordinals are unique) -- but its not sorting unique ordinals…

> The (sequential) distribution of digits amongst sorted numbers is tiny

This is why 10^80 random lists gets reduced to only 10^36 sorted lists. However, 10^36 is still very large with respect to the size of the model.

Re: From word models to world models

#105

Post by mjburgess »

jbay808 wrote: post #104
mjburgess wrote: post #100

Earlier quoted context omitted.

> Can you provide more details about what you mean by this distributional structure The distribution of sorted digits is: (0 1 2 3 4 5 6 7 8 9) before (1 before 0 1 2 3 4 5 6 7 8 9) before (2 before 0 1 2 3 4 5 6 7 8 9) before (3 before 0 1 2 3 4 5 6 7 8 9) ... ... When you compute the search space you're treating each number as a unique token (ie., that all ordinals are unique) -- but its not sorting unique ordinals…

> The (sequential) distribution of digits amongst sorted numbers is tiny This is why 10^80 random lists gets reduced to only 10^36 sorted lists. However, 10^36 is still very large with respect to the size of the model.

You're treating each list as unique, all the lists have a distribution of digits in common... I'm at a loss to even understand what you're saying here really -- this is why you need to actually state, formally, what you think the "LLMs are just stats" hypothesis amounts to.

It seems you think it amounts to saying LLMs sample from a combinatorial space, naively construed -- but that isnt the claim?

The claim is rather, they sample from a statistical distribution of tokens.

Take each position in the input vector, 1...127. It needs to "learn":

P(x0 position | y, x1...x127 positions), P(1|y, 2...127), P(2|y, 3...127), etc.

Which is a family of 127 conditional distributions that seem trivial to learn.

I really don't know why you think the size of a combinatorial space is relevant here?

All the sorted lists share basically the same tiny family of conditional distributions { P(x_i | x_(i-1)...x_127) }

Re: From word models to world models

#106

Post by [deleted] »

jbay808 wrote: post #101
YeGoblynQueenne wrote: post #94

Earlier quoted context omitted.

Your model is not sorting correctly and it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules…

> it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). > Also, Machine Learning 101: you test your models on a test set that is disjoint to…

[deleted]

Re: From word models to world models

#107

Post by YeGoblynQueenne »

jbay808 wrote: post #102
YeGoblynQueenne wrote: post #95

Earlier quoted context omitted.

I'm curious, why are you using "n-gram" as if you're referring to a model? You say e.g. that "LLM is learning an n-gram". N-grams are features, not models. You can train an n-gram model, or you can train a language model using n-grams as features, and so on, but you can't "learn an n-gram". Where did you find this terminology? EDIT: >> and Hyp2 is that the LLM's training molds it into representing an actual sorting a…

> what do you think would happen if you gave your model an alphanumeric list to sort? Did you try that? The tokenizer would throw an exception, because it doesn't have any tokens to represent alphabetical characters. But you tell me - if I had tokenized alphabetical characters and defined an ordering, would you expect the results to be any different? > You say e.g. that "LLM is learning an n-gram"[...] you can't "lea…

>> Where do I say that?

In your comment above:

(...) is expressed a little bit more clearly as _the LLM is learning an n-gram_ that produces correct sorts (...)

(My underlining)

You also use it in a similarly unusual way throughout your linked substack post, for example, you write:

the way GPT works is, in a certain sense, functionally equivalent to an n-gram, but that doesn’t mean GPT is an n-gram.

Where does this use of "n-gram" come from? I mean, did you see it somewhere? I'm curious, where?

>> The tokenizer would throw an exception, because it doesn't have any tokens to represent alphabetical characters. But you tell me - if I had tokenized alphabetical characters and defined an ordering, would you expect the results to be any different?

I'm sorry, I don't understand. "Defined an ordering", where?

You can change your tokenizer but that will not change the trained model, obviously. So if you take your model that's trained on two-digit lists of integers and you run it on lists of any other type of elements it will not be able to sort them correctly. But isn't that what you claim? That:

"the LLM's training molds it into representing an actual sorting algorithm that would correctly generalize to any input list"

"Any input list"? How so?

Re: From word models to world models

#108

Post by YeGoblynQueenne »

jbay808 wrote: post #101
YeGoblynQueenne wrote: post #94

Earlier quoted context omitted.

Your model is not sorting correctly and it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules…

> it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). > Also, Machine Learning 101: you test your models on a test set that is disjoint to…

>> Transformers are RASP programs, which includes sorting programs. See the Weiss paper (https://arxiv.org/pdf/2106.06981.pdf).

That's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP". It's not any kind of established terminology, nor is it anything but a claim.

What is certainly established is that a function, and an algorithm, are different objects. To clarify, a function is a mapping between the elements of two sets, whereas an algorithm is a sequence of operations that calculates the result of a function and is guaranteed to terminate. Algorithms are also typically understood to be provably correct and to have some provable asymptotic complexity (as opposed, for example, to heuristics) but that's not a requirement.

So for example, if you have a function ƒ between sets X and Y, and an algorithm P that calculates the result of ƒ, then you can give any element of X to P and it will return (in fact, construct) an element of Y. Crucially, ƒ is not P, and P is not ƒ.

Now, when you train a machine learning model, you are typically training a function ƒ̂ (with a little hat) to approximate ƒ. That means that your trained ƒ̂ is a function that maps some of the elements of X to the same elements of Y as ƒ, but not all. It's an approximation. So you get some amount of error, as in your experiment.

So what you've done in your experiment is that you trained a model to approximate a mapping between the set of lists, to itself (where the input list is any of the lists in your training set and the output is the same list, sorted). Your model is not an algorithm, and you cannot train an algorithm with a language model.

I appreciate that, learning an algorithm, is what you wanted to achieve, but in science we don't choose the answer that pleases us, we choose the answer that makes the most sense- and a good heuristic for that is that the answer that makes more sense is the simplest one. Here, in order to convince yourself that you have trained a language model to learn an algorithm, rather than an approximator, you have chosen to rely on a preprint with a completely novel and untested concept that someone put on the internet, rather than the well-understood abstractions of elementary computer science, so not at all the simplest explanation. That is not a good idea. You will not understand what is going on, if you rely on that kind of explanation. I assume you are trying to understand?

Edit: incidentally, you don't need a transformer to train an approximator to a sorting function. You can do that with a multi-layer perceptron, or a logistic regression, certainly with an LSTM. Ceteris paribus, you'll get the same results.

>> The probability of a test list existing in the training set is less than 10^-70.

But the same probability if you held the test set out would be 0, so why not do that? It's not hard to do.

Is there a good reason not to do that?

Btw, lists are composite objects. How much overlap is there between your training and test lists? Do you know?

Edit: meh. HN messes up my nice f-with-hook-and-combining-circumflex-accent. DAAAAANG!!!!

Re: From word models to world models

#109

Post by [deleted] »

jbay808 wrote: post #101
YeGoblynQueenne wrote: post #94

Earlier quoted context omitted.

Your model is not sorting correctly and it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules…

> it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners. Transformers are RASP programs, which includes sorting programs. See the Weiss paper ( https://arxiv.org/pdf/2106.06981.pdf ). > Also, Machine Learning 101: you test your models on a test set that is disjoint to…

[deleted]

Re: From word models to world models

#110

Post by gibsonf1 »

YeGoblynQueenne wrote: post #97
drdeca wrote: post #90

Earlier quoted context omitted.

What's a concept?

In general, in AI, when we talk about "concepts" we're talking about the things machine learning models are trained to, well, to model. In PAC-Learning terms, specifically, a "concept" is a set of instances (which may be vectors or whatever). Note that a "concept" is not the same as a "class", as in classification. Instead a concept belongs to a class of similar concepts and a learner is trained on instances of conce…

I think modeling a human concept would be a far better approach.
Post reply on HN