Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs aren't world models

yosefk.com

Post reply on HN
Print view View original

51–60 of 240 posts

Re: LLMs aren't world models

#51

Post by WillPostForFood »

yosefk wrote: post #6
libraryofbabel wrote: post #4

This essay could probably benefit from some engagement with the literature on “interpretability” in LLMs, including the empirical results about how knowledge (like addition) is represented inside the neural network. To be blunt, I’m not sure being smart and reasoning from first principles after asking the LLM a lot of questions and cherry picking what it gets wrong gets to any novel insights at this point. And it alr…

Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very u…

Your LLM output seems abnormally bad, like you are using old models, bad models, or intentionally poor prompting. I just copied and pasted your Krita example into ChatGPT, and reasonable answer, nothing like what you paraphrased in your post.

https://imgur.com/a/O9CjiJY

Re: LLMs aren't world models

#52

Post by rpdillon »

jonplackett wrote: post #46
RugnirViking wrote: post #31

Earlier quoted context omitted.

this seems like a strange riddle. In my mind I was thinking that regardless of the glass, all of the objects can be seen (due to perspective, and also the fact you mentioned the locations, meaning you're aware of them). It seems to me it would only actually work in an orthographic perspective, which is not how our reality works

You can tell from the response it does understand the riddle just fine, it just gets it wrong.

Have you asked five adults this riddle? I suspect at least two of them would get it wrong or have some uncertainty about whether or not the peanut was visible.

Re: LLMs aren't world models

#53

Post by rpdillon »

yosefk wrote: post #3
t0md4n wrote: post #2

https://arxiv.org/abs/2501.17186

This is interesting. The "professional level" rating of However: "A significant Elo rating jump occurs when the model’s Legal Move accuracy reaches 99.8%. This increase is due to the reduction in errors after the model learns to generate legal moves, reinforcing that continuous error correction and learning the correct moves significantly improve ELO" You should be able to reach the move legality of around 100% with…

> Failing to do so means that it has not learned a model of what chess is, at some basic level.

I'm not sure about this. Among a standard amateur set of chess players, how often when they lack any kind of guidance from a computer do they attempt to make a move that is illegal? I played chess for years throughout elementary, middle and high school, and I would easily say that even after hundreds of hours of playing, I might make two mistakes out of a thousand moves where the move was actually illegal, often because I had missed that moving that piece would continue to leave me in check due to a discovered check that I had missed.

It's hard to conclude from that experience that players that are amateurs lack even a basic model of chess.

Re: LLMs aren't world models

#54

Post by williamcotton »

AyyEye wrote: post #8
libraryofbabel wrote: post #4

This essay could probably benefit from some engagement with the literature on “interpretability” in LLMs, including the empirical results about how knowledge (like addition) is represented inside the neural network. To be blunt, I’m not sure being smart and reasoning from first principles after asking the LLM a lot of questions and cherry picking what it gets wrong gets to any novel insights at this point. And it alr…

With LLMs being unable to count how many Bs are in blueberry, they clearly don't have any world model whatsoever. That addition (something which only takes a few gates in digital logic) happens to be overfit into a few nodes on multi-billion node networks is hardly a surprise to anyone except the most religious of AI believers.

I don’t solve math problems with my poetry writing skills:

https://chatgpt.com/share/689ba837-8ae0-8013-96d2-7484088f27...

Re: LLMs aren't world models

#55

Post by xg15 »

rpdillon wrote: post #52
jonplackett wrote: post #46

Earlier quoted context omitted.

You can tell from the response it does understand the riddle just fine, it just gets it wrong.

Have you asked five adults this riddle? I suspect at least two of them would get it wrong or have some uncertainty about whether or not the peanut was visible.

This. Was also thinking "yes" first because of the glass of water, transparency, etc, but then got unsure: The objects might be spaced so widely that the milk or coke bottle would obscure the view due to perspective - or the peanut would simply end up outside the viewer's field of vision.

Shows that even if you have a world model, it might not be the right one.

Re: LLMs aren't world models

#57

Post by geraneum »

keeda wrote: post #39

That whole bit about color blending and transparency and LLMs "not knowing colors" is hard to believe. I am literally using LLMs every day to write image-processing and computer vision code using OpenCV. It seamlessly reasons across a range of concepts like color spaces, resolution, compression artifacts, filtering, segmentation and human perception. I mean, removing the alpha from a PNG image was a preprocessing ste…

> it wrote by itself as part of a larger task I had given it, so it certainly understands transparency

Or it’s a common step or a known pattern or combination of steps that is prevalent in its training data for certain input. I’m guessing you don’t know what’s exactly in the training sets. I don’t know either. They don’t tell ;)

> but it adapts and combines them well to suit my particular requirements. So they seem to handle "novel" pretty well too.

We tend to overestimate the novelty of our own work and our methods and at the same time, underestimate the vastness of the data and information available online for machines to train on. LLMs are very sophisticated pattern recognizers. It doesn’t mean what you are doing specifically is done in this exact way before, rather the patterns adapted and the approach may not be one of their kind.

> is a purely philosophical question

It is indeed. A question we need to ask ourselves.

Re: LLMs aren't world models

#58

Post by marcellus23 »

WillPostForFood wrote: post #51
yosefk wrote: post #6

Earlier quoted context omitted.

Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very u…

Your LLM output seems abnormally bad, like you are using old models, bad models, or intentionally poor prompting. I just copied and pasted your Krita example into ChatGPT, and reasonable answer, nothing like what you paraphrased in your post. https://imgur.com/a/O9CjiJY

I think it's hard to take any LLM criticism seriously if they don't even specify which model they used. Saying "an LLM model" is totally useless for deriving any kind of conclusion.

Re: LLMs aren't world models

#59

Post by ameliaquining »

One thing I appreciated about this post, unlike a lot of AI-skeptic posts, is that it actually makes a concrete falsifiable prediction; specifically, "LLMs will never manage to deal with large code bases 'autonomously'". So in the future we can look back and see whether it was right.

For my part, I'd give 80% confidence that LLMs will be able to do this within two years, without fundamental architectural changes.

Re: LLMs aren't world models

#60

Post by p1esk »

marcellus23 wrote: post #58
WillPostForFood wrote: post #51

Earlier quoted context omitted.

Your LLM output seems abnormally bad, like you are using old models, bad models, or intentionally poor prompting. I just copied and pasted your Krita example into ChatGPT, and reasonable answer, nothing like what you paraphrased in your post. https://imgur.com/a/O9CjiJY

I think it's hard to take any LLM criticism seriously if they don't even specify which model they used. Saying "an LLM model" is totally useless for deriving any kind of conclusion.

Yes, I’d be curious about his experience with GPT-5 Thinking model. So far I haven’t seen any blunders from it.
Post reply on HN