Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs aren't world models

yosefk.com

Post reply on HN
Print view View original

81–90 of 240 posts

Re: LLMs aren't world models

#81

Post by slt2021 »

ameliaquining wrote: post #59

One thing I appreciated about this post, unlike a lot of AI-skeptic posts, is that it actually makes a concrete falsifiable prediction; specifically, "LLMs will never manage to deal with large code bases 'autonomously'". So in the future we can look back and see whether it was right. For my part, I'd give 80% confidence that LLMs will be able to do this within two years, without fundamental architectural changes.

>LLMs will never manage to deal

time to prove hypothesis: infinity years

Re: LLMs aren't world models

#82

Post by pron »

bithive123 wrote: post #62

Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that w…

> Symbols, by definition, only represent a thing. They are not the same as the thing

First of all, the point isn't about the map becoming the territory, but about whether LLMs can form a map that's similar to the map in our brains.

But to your philosophical point, assuming there are only a finite number of things and places in the universe - or at least the part of which we care about - why wouldn't they be representable with a finite set of symbols?

What you're rejecting is the Church-Turing thesis [1] (essentially, that all mechanical processes, including that of nature, can be simulated with symbolic computation, although there are weaker and stronger variants). It's okay to reject it, but you should know that not many people do (even some non-orthodox thoughts by Penrose about the brain not being simulatable by an ordinary digital computer still accept that some physical machine - the brain - is able to represent what we're interested in).

> If we accept the incompleteness theorem

There is no if there. It's a theorem. But it's completely irrelevant. It means that there are mathematical propositions that can't be proven or disproven by some system of logic, i.e. by some mechanical means. But if something is in the universe, then it's already been proven by some mechanical process: the mechanics of nature. That means that if some finite set of symbols could represent the laws of nature, then anything in nature can be proven in that logical system. Which brings us back to the first point: the only way the mechanics of nature cannot be represented by symbols is if they are somehow infinite, i.e. they don't follow some finite set of laws. In other words - there is no physics. Now, that may be true, but if that's the case, then AI is the least of our worries.

Of course, if physics does exist - i.e. the universe is governed by a finite set of laws - that doesn't mean that we can predict the future, as that would entail both measuring things precisely and simulating them faster than their operation in nature, and both of these things are... difficult.

[1]: https://plato.stanford.edu/entries/church-turing/

Re: LLMs aren't world models

#83

Post by [deleted] »

yosefk wrote: post #26
libraryofbabel wrote: post #22

Earlier quoted context omitted.

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

I'm not saying that LLMs can't learn about the world - I even mention how they obviously do it, even at the learned embeddings level. I'm saying that they're not compelled by their training objective to learn about the world and in many cases they clearly don't, and I don't see how to characterize the opposite cases in a more useful way than "happy accidents." I don't really know how they are made "good at math," and…

[deleted]

Re: LLMs aren't world models

#84

Post by pron »

chamomeal wrote: post #75
bithive123 wrote: post #62

Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that w…

I’m not a math guy but the incompleteness theorem applies to formal systems, right? I’ve never thought about LLMs as formal systems, but I guess they are?

Anything that runs on a computer is a formal system. "Formal" (the manipulation of forms) is an old term for what, after Turing, we call "mechanical".

Re: LLMs aren't world models

#85

Post by wizzwizz4 »

shinycode wrote: post #72
ameliaquining wrote: post #59

One thing I appreciated about this post, unlike a lot of AI-skeptic posts, is that it actually makes a concrete falsifiable prediction; specifically, "LLMs will never manage to deal with large code bases 'autonomously'". So in the future we can look back and see whether it was right. For my part, I'd give 80% confidence that LLMs will be able to do this within two years, without fundamental architectural changes.

« autonomously » what happens when subtle updates that are not bugs but change the meaning of some features that might break the workflow on some other external parts of a client’s system ? It happens all the time and, because it’s really hard to have the whole meaning and business rules written and maintained up to date, an LLM might never be able to grasp some meaning. Maybe if instead of developing code and infras…

Those spec sheets exist: they're called software.

Re: LLMs aren't world models

#86

Post by calf »

yosefk wrote: post #26
libraryofbabel wrote: post #22

Earlier quoted context omitted.

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

I'm not saying that LLMs can't learn about the world - I even mention how they obviously do it, even at the learned embeddings level. I'm saying that they're not compelled by their training objective to learn about the world and in many cases they clearly don't, and I don't see how to characterize the opposite cases in a more useful way than "happy accidents." I don't really know how they are made "good at math," and…

But this is parallel to saying LLMs are not "compelled" by the training algorithms to learn symbolic logic.

Which says to me there are two camps on this and the verdict is still out on this and all related questions.

Re: LLMs aren't world models

#87

Post by bithive123 »

auggierose wrote: post #76
bithive123 wrote: post #74

Earlier quoted context omitted.

I knew someone would call me out on that. I used the wrong word; what I meant was "expressed in a way that would satisfy" which implies proof within the symbolic order being used. I don't claim to be a mathematician or philosopher.

Well, you don't get it. The LLM definitely can state propositions "that satisfy", let's just call them true propositions, and that this is not the same as having a proof for it is what the incompleteness theorem says. Why would you require an LLM to have proof for the things it says? I mean, that would be nice, and I am actually working on that, but it is not anything we would require of humans and/or HN commenters,…

I clearly do not meet the requirements to use the analogy.

I am hearing the term super intelligence a lot and it seems to me the only form that would take is the machine spitting out a bunch of symbols which either delight or dismay the humans. Which implies they already know what it looks like.

If this technology will advance science or even be useful for everyday life, then surely the propositions it generates will need to hold up to reality, either via axiomatic rigor or empirically. I look forward to finding out if that will happen.

But it's still just a movement from the known to the known, a very limited affair no matter how many new symbols you add in whatever permutation.

Re: LLMs aren't world models

#88

Post by astrange »

pron wrote: post #82
bithive123 wrote: post #62

Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that w…

> Symbols, by definition, only represent a thing. They are not the same as the thing First of all, the point isn't about the map becoming the territory, but about whether LLMs can form a map that's similar to the map in our brains. But to your philosophical point, assuming there are only a finite number of things and places in the universe - or at least the part of which we care about - why wouldn't they be represent…

> First of all, the point isn't about the map becoming the territory, but about whether LLMs can form a map that's similar to the map in our brains.

It should be capable of something similar (fsvo similar), but the largest difference is that humans have to be power-efficient and LLMs do not.

That is, people don't actually have world models, because modeling something is a waste of time and energy insofar as it's not needed for anything. People are capable of taking out the trash without knowing what's in the garbage bag.

Re: LLMs aren't world models

#89

Post by bevr1337 »

scarmig wrote: post #79
bithive123 wrote: post #62

Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that w…

> If we accept the incompleteness theorem And, by various universality theorems, a sufficiently large AGI could approximate any sequence of human neuron firings to an arbitrary precision. So if the incompleteness theorem means that neural nets can never find truth, it also means that the human brain can never find truth. Human neuron firing patterns, after all, only represent a thing; they are not the same as the thi…

> And, by various universality theorems, a sufficiently large AGI could approximate any sequence of human neuron firings to an arbitrary precision.

Wouldn't it become harder to simulate a human brain the larger a machine is? I don't know nothing, but I think that peaky speed of light thing might pose a challenge.

Re: LLMs aren't world models

#90

Post by astrange »

yosefk wrote: post #9
AyyEye wrote: post #8

Earlier quoted context omitted.

With LLMs being unable to count how many Bs are in blueberry, they clearly don't have any world model whatsoever. That addition (something which only takes a few gates in digital logic) happens to be overfit into a few nodes on multi-billion node networks is hardly a surprise to anyone except the most religious of AI believers.

Actually I forgive them those issues that stem from tokenization. I used to make fun at them for listing datum as a noun whose plural form ends with an i, but once I learned about how tokenization works, I no longer do it - it feels like mocking a person's intelligence because of a speech impediment or something... I am very kind to these things, I think

Tokenization makes things harder, but it doesn't make them impossible. Just takes a bit more memorization.

Other writing systems come with "tokenization" built in making it still a live issue. Think of answering:

1. How many n's are in 日本?

2. How many ん's are in 日本?

(Answers are 2 and 1.)

Post reply on HN