Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From word models to world models

arxiv.org

Post reply on HN
Print view View original

91–100 of 119 posts

Re: From word models to world models

#91

Post by canjobear »

gibsonf1 wrote: post #89
canjobear wrote: post #87

Earlier quoted context omitted.

Read more carefully. Their "language of thought" is not a natural language, it's a variant of lambda calculus with probabilistic semantics.

Right, derived from word pattern statistics. The CYC project tried first order predicate calculus with complete failure. This is not how we think or how conceptual awareness works. The key give away is what they don't talk about, Concepts.

They talk about concepts several times, defined as probabilistic functions within their LOT.

Re: From word models to world models

#92

Post by taliesinb »

gibsonf1 wrote: post #50
buzzy_hacker wrote: post #36

Earlier quoted context omitted.

Maybe they can’t be so fully separated. https://en.m.wikipedia.org/wiki/Linguistic_relativity

I think the key point is that serialized words symbolize concepts and other logic such that if you can't retrieve that concept into your awareness, you will not understand the word. Learning and forming the concepts comes prior to attaching common word symbols to them based on the region you live in. So if you start with words, you never get anywhere, hence the complete lack of any intelligence in the LLM approach.

Exactly. Thought is prior to language, and much confusion happens when you conflate the them. In particular the surface syntax of language tells you next to nothing about the "syntax" of thought, which is hypergraphical, not tree-structured.

Re: From word models to world models

#93

Post by YeGoblynQueenne »

famouswaffles wrote: post #19
mjburgess wrote: post #16

Earlier quoted context omitted.

> Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial l…

>Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?

>> alright let me humor you for a bit.

That's real class, right there.

Re: From word models to world models

#94

Post by YeGoblynQueenne »

jbay808 wrote: post #25
mjburgess wrote: post #3

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…

> It is absolutely trivial to show Hyp2 is false To investigate precisely this question in a clear and unambiguous way, I trained an LLM from scratch to sort lists of numbers. It learned to sort them correctly, and the entropy is such that it's absolutely impossible that it could have done this by Hyp1 (sampling from similar text in the training set). https://jbconsulting.substack.com/p/its-not-just-statistics-... No…

Your model is not sorting correctly and it sure has not learned any "algorithm". At best it has learned to approximate a sorting algorithm. That's what statistical machine learning models do, they are function approximators; not program learners.

Also, Machine Learning 101: you test your models on a test set that is disjoint to the training set. To clarify, we do this not because it's in the book and that's the rules, but because, by testing the model on held-out data, we can predict the error the model will have on unseen data (i.e. data not available to the experimenter). And we do this because under PAC-Learning assumptions a learner is said to learn a concept when it can correctly label instances of the concept with some probability of some error. In real-world situations we do not know the true concept, so we test on held-out data to approximate the probability of error.

Bottom line, if you train a model to do a thing and you don't test it carefully to figure out its error, you might claim it's learned something, but in truth, you have no idea what it's learned.

(To clarify: you tested on the train data assuming there's a low probability of overlap. Don't do that if you're trying to understand what your models can do).

Re: From word models to world models

#95

Post by YeGoblynQueenne »

jbay808 wrote: post #37
mjburgess wrote: post #30

Earlier quoted context omitted.

So this is a really good starting point -- but you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something". Formally, what hypotheses are you comparing? What do you think the specific hypothesis of the "AI = stats" person is? It isnt that the NN literally remembers data tokens, right? In any case: The issue with forcing NNs to model mathematical features is that t…

> you havent formulated any hypotheses that can be tested. You've just looked at the graph and "reckoned something" Let's not be so hasty. I think I do put it as clearly as possible. I'm comparing essentially your Hyp1 and Hyp2, where Hyp1 (aka the stochastic parrot ) is expressed a little bit more clearly as the LLM is learning an n -gram that produces correct sorts through rote memorization of statistical correlati…

I'm curious, why are you using "n-gram" as if you're referring to a model? You say e.g. that "LLM is learning an n-gram". N-grams are features, not models. You can train an n-gram model, or you can train a language model using n-grams as features, and so on, but you can't "learn an n-gram".

Where did you find this terminology?

EDIT:

>> and Hyp2 is that the LLM's training molds it into representing an actual sorting algorithm that would correctly generalize to any input list.

Btw, you have not shown anything like that. You trained and tested on lists of two-digit positive integers expressible in 128 characters. That's not "any input list". As a for instance, what do you think would happen if you gave your model an alphanumeric list to sort? Did you try that?

Your model also doesn't correctly generalise, not even to its own training set that you tested it on. There's plenty of error in the figure where you show its accuracy (not clear if that's training or test accuracy).

It's not clear to me how you account for those obvious limitations of your model (it's a toy model after all) when you claim that it "learned to implement a sorting algorithm" etc. It would be great if you could clarify that.

Re: From word models to world models

#96

Post by YeGoblynQueenne »

jstanley wrote: post #55
mjburgess wrote: post #51

Earlier quoted context omitted.

> that's no guarantee that the LLM will be able to compress it efficiently Your LLM here is 600MB which is a grossly inefficient compression of the sort space. If LLMs "learned algorithms", the best compression would be on the order of bytes. The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it! What do you think all those MBs are doing? They're the extraordinary cost of the "stati…

> The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it! This isn't a fair comparison. The python code to sort a list is leveraging an enormous amount of information that is stored outside the python code, whereas the GPT version basically has to do it "from scratch", and in a very convoluted computing model. A better comparison would be "how many bits does it take to encode a confi…

The model the OP trained also runs on a computer. If I'm not mistaken, it also runs on Python itself.

Which means you don't need to count all the bytes in the infrastructure all the way down to the electric grid, maybe. You can compare a sorting algorithm to a sorting model, as stand-alone programs, on their relative size, and that will give you a good idea of how much work each is doing.

Re: From word models to world models

#97

Post by YeGoblynQueenne »

drdeca wrote: post #90
gibsonf1 wrote: post #89

Earlier quoted context omitted.

Right, derived from word pattern statistics. The CYC project tried first order predicate calculus with complete failure. This is not how we think or how conceptual awareness works. The key give away is what they don't talk about, Concepts.

What's a concept?

In general, in AI, when we talk about "concepts" we're talking about the things machine learning models are trained to, well, to model.

In PAC-Learning terms, specifically, a "concept" is a set of instances (which may be vectors or whatever).

Note that a "concept" is not the same as a "class", as in classification. Instead a concept belongs to a class of similar concepts and a learner is trained on instances of concepts in a class. Then a learner is said to be capable of learning the concepts in a class if it can correctly label instances of a concept in the class with some probability of some error.

For a more concrete example, a "class" of concepts is the class of objects represented as subsets of pixels in digital images. A "concept" of that class is, for example, the concept "dog". An image classifier can be said to be able to learn to identify objects in images if it can correctly classify subsets of the pixels in an image as "dog" (or "not dog").

Since the article above is coming from Josh Tenenbaum's group, that's the kind of terminology you should have in mind, when you're talking about "concepts". These guys are old-school (and I say that as a compliment).

Re: From word models to world models

#98

Post by anankaie »

bryan0 wrote: post #61
mjburgess wrote: post #3

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…

Failing on "trivial alterations to the same underlying domain" is a not a disproof of thought. Your argument also implies hyp1 and 2 are exclusive, clearly both can be true, and in fact must be true, unless you are claiming that you do not "sample" from similar language to express your own thoughts? Where does your language come from then, if not learning from previous experience?

While I agree with you on the relation of GP's Hyp1 and Hyp2, you are making an unfounded assumption of a sampling process being necessary to perform human speech. I do not believe we have the understanding of how thought is represented in the human brain to make that judgement. In other words, just because sampling from a distribution can produce human-like text does not mean that it is the only way to do that, and thus that it must be the way that humans produce text, spoken or written.

Re: From word models to world models

#99

Post by bryan0 »

anankaie wrote: post #98
bryan0 wrote: post #61

Earlier quoted context omitted.

Failing on "trivial alterations to the same underlying domain" is a not a disproof of thought. Your argument also implies hyp1 and 2 are exclusive, clearly both can be true, and in fact must be true, unless you are claiming that you do not "sample" from similar language to express your own thoughts? Where does your language come from then, if not learning from previous experience?

While I agree with you on the relation of GP's Hyp1 and Hyp2, you are making an unfounded assumption of a sampling process being necessary to perform human speech. I do not believe we have the understanding of how thought is represented in the human brain to make that judgement. In other words, just because sampling from a distribution can produce human-like text does not mean that it is the only way to do that, and…

We might be talking about 2 different things. I was referring to the backwards learning pass and you seem to be referring to the forward inference pass, but what is an alternative to learning (or producing) text which does not involve sampling from some larger space? (Also I’m not a statistician so I’m not sure if these are technically “distributions”)

Re: From word models to world models

#100

Post by mjburgess »

jbay808 wrote: post #68
mjburgess wrote: post #51

Earlier quoted context omitted.

> that's no guarantee that the LLM will be able to compress it efficiently Your LLM here is 600MB which is a grossly inefficient compression of the sort space. If LLMs "learned algorithms", the best compression would be on the order of bytes. The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it! What do you think all those MBs are doing? They're the extraordinary cost of the "stati…

> If LLMs "learned algorithms", the best compression would be on the order of bytes. Yes. Except: (1) the model size is fixed during training, it would be impossible to obtain a bytes-sized result regardless of what it learns to represent. One might even open the thing up and find bubblesort* inside followed by 599 MB of junk DNA; that size is dictated by how it was initialized. (2) I'm not claiming this model is a m…

> Can you provide more details about what you mean by this distributional structure

The distribution of sorted digits is:

(0 1 2 3 4 5 6 7 8 9) before

(1 before 0 1 2 3 4 5 6 7 8 9) before

(2 before 0 1 2 3 4 5 6 7 8 9) before

(3 before 0 1 2 3 4 5 6 7 8 9) ...

...

When you compute the search space you're treating each number as a unique token (ie., that all ordinals are unique) -- but its not sorting unique ordinals, it's sorting digits in a sequential model ie., it learns P(Next|Prev)

The (sequential) distribution of digits amongst sorted numbers is tiny

Post reply on HN