Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs aren't world models

yosefk.com

Post reply on HN
Print view View original

131–140 of 240 posts

Re: LLMs aren't world models

#131

Post by jijijijij »

libraryofbabel wrote: post #23
andyjohnson0 wrote: post #16

Earlier quoted context omitted.

> With LLMs being unable to count how many Bs are in blueberry, they clearly don't have any world model whatsoever. Is this a real defect, or some historical thing? I just asked GPT-5: How many "B"s in "blueberry"? and it replied: There are 2 — the letter b appears twice in "blueberry". I also asked it how many Rs in Carrot, and how many Ps in Pineapple, amd it answered both questions correctly too.

It’s a historical thing that people still falsely claim is true, bizarrely without trying it on the latest models. As you found, leading LLMs don’t have a problem with it anymore.

The question is, did these LLMs figured it out by themselves or has someone programmed a specific coroutine to address this „issue“, to make it look smarter than it is?

On a trillion dollar budget, you could just crawl the web for AI tests people came up with and solve them manually. We know it‘s a massively curated game. With that kind of money you can do a lot of things. You could feed every human on earth countless blueberries for starters.

Calling an algorithm to count letters in a word isn’t exactly worth the hype tho is it?

The point is, we tend to find new ways these LLMs can’t figure out the most basic shit about the world. Horses can count. Counting is in everything. If you read every text ever written and still can’t grasp counting you simply are not that smart.

Re: LLMs aren't world models

#132

Post by jijijijij »

Terr_ wrote: post #101
AyyEye wrote: post #41

Earlier quoted context omitted.

They clearly RLHF out the embarrassing cases and make cheating on benchmarks into a sport.

I wouldn't be surprised if some models get set up to identify that type of question and run the word through string processing function.

Of course they do stuff like that, otherwise it would look like they are stagnating. Fake it till you make it. Tho, at this point, the world is in deep shit, if they don’t make it…

Re: LLMs aren't world models

#133

Post by 2muchcoffeeman »

libraryofbabel wrote: post #22
yosefk wrote: post #6

Earlier quoted context omitted.

Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very u…

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

It’s not just on the job learning though. I’m no AI expert, but the fact that you have “prompt engineers” and AI doesn’t know what it doesn’t know, gives me pause.

If you ask an expert, they know the bounds of their knowledge and can understand questions asked to them in multiple ways. If they don’t know the answer, they could point to someone who does or just say “we don’t know”.

LLMs just lie to you and we call it “hallucinating“ as though they will eventually get it right when the drugs wear off.

Re: LLMs aren't world models

#134

Post by cavisne »

smnrchrds wrote: post #97
jononor wrote: post #96

Earlier quoted context omitted.

"LLM" as well, because coding agents are already more than just an LLM. There is very useful context management around it, and tool calling, and ability to run tests/programs, etc. Though they are LLM-based systems, they are not LLMs.

Indeed. If the LLM calls a chess engine tool behind the scenes, it would be able to play excellent chess as well.

The author would still be wrong in the tool-calling scenario. There is already perfect (or at least superhuman) chess engines. There is no perfect "coding engine". LLM's + tools being able to reliably work on large codebases would be a new thing.

Re: LLMs aren't world models

#135

Post by eru »

simiones wrote: post #48
libraryofbabel wrote: post #38

Earlier quoted context omitted.

> they clearly don't have any world model whatsoever Then how did an LLM get gold on the mathematical Olympiad, where it certainly hadn’t seen the questions before? How on earth is that possible without a decent working model of mathematics? Sure, LLMs might make weird errors sometimes (nobody is denying that), but clearly the story is rather more complicated than you suggest.

> where it certainly hadn’t seen the questions before? What are you basing this certainty on? And even if you're right that the specific questions had not come up, it may still be that the questions from the math olympiad were rehashes of similar questions in other texts, or happened to correspond well to a composition of some other problems that were part of the training set, such that the LLM could 'pick up' on the…

> What are you basing this certainty on?

People make up new questions for each IMO.

Re: LLMs aren't world models

#136

Post by joe_the_user »

libraryofbabel wrote: post #4

This essay could probably benefit from some engagement with the literature on “interpretability” in LLMs, including the empirical results about how knowledge (like addition) is represented inside the neural network. To be blunt, I’m not sure being smart and reasoning from first principles after asking the LLM a lot of questions and cherry picking what it gets wrong gets to any novel insights at this point. And it alr…

I think both the literature on interpretability and explorations on internal representations actually reinforce the author's conclusion. I think internal representation research tends to nets that deal with a single "model" don't necessary have the same representation and don't necessarily have a single representation.

And doing well on XYZ isn't evidence of a world model in particular. The point that these things aren't always using a world is reinforced by systems being easily confused by extraneous information, even systems as sophisticated as thus that can solve Math Olympiad questions. The literature has said "ad-hoc predictors" for a long time and I don't think much has changed - except things do better on benchmarks.

And, humans too can act without a consistent world model.

Re: LLMs aren't world models

#137

Post by eru »

libraryofbabel wrote: post #22
yosefk wrote: post #6

Earlier quoted context omitted.

Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very u…

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

> A lot of that success is from reinforcement learning techniques where the LLM is made to solve tons of math problems after the pre-training “read everything” step, which then gives it a chance to update its weights. LLMs aren’t just trained from reading a lot of text anymore. It’s very similar to how the alpha zero chess engine was trained, in fact.

It's closer to AlphaGo, which first trained on expert human games and then 'fine tuned' with self-play.

AlphaZero specifically did not use human training data at all.

I am waiting for an AlphaZero style general AI. ('General' not in the GAI sense but in the ChatGPT sense of something you can throw general problems at and it will give it a good go, but not necessarily at human level, yet.) I just don't want to call it an LLM, because it wouldn't necessarily be trained on language.

What I have in mind is something that first solves lots and lots of problems, eg logic problems, formally posed programming problems, computer games, predicting of next frames in a web cam video, economic time series, whatever, as a sort-of pre-training step and then later perhaps you feed it a relatively small amount of human readable text and speech so you can talk to it.

Just to be clear: this is not meant as a suggestion for how to successfully train an AI. I'm just curious whether it would work at all and how well / how badly.

Presumably there's a reason why all SOTA models go 'predict human produced text first, then learn problem solving afterwards'.

> I think the biggest limitation LLMs actually have, the one that is the biggest barrier to AGI, is that they can’t learn on the job, during inference. This means that with a novel codebase they are never able to build a good model of it, because they can never update their weights. [...]

Yes, I agree. But 'on-the-job' training is also such an obvious idea that plenty of people are working on making it work.

Re: LLMs aren't world models

#138

Post by eru »

2muchcoffeeman wrote: post #133
libraryofbabel wrote: post #22

Earlier quoted context omitted.

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

It’s not just on the job learning though. I’m no AI expert, but the fact that you have “prompt engineers” and AI doesn’t know what it doesn’t know, gives me pause. If you ask an expert, they know the bounds of their knowledge and can understand questions asked to them in multiple ways. If they don’t know the answer, they could point to someone who does or just say “we don’t know”. LLMs just lie to you and we call it…

> I’m no AI expert, but the fact that you have “prompt engineers” [...] gives me pause.

Why? A bunch of human workers can get a lot more done with a capable leader who helps prompt them in the right direction and corrects oversights etc.

And overall, prompt engineering seems like exactly the kind of skill AI will be able to develop by itself. You already have a bit like this happening: when you ask Gemini to create a picture for you, then the language part of Gemini will take your request and engineer a prompt for the picture part of Gemini.

Re: LLMs aren't world models

#139

Post by ehnto »

eru wrote: post #125
typpilol wrote: post #63

Earlier quoted context omitted.

This seems like a common theme with these types of articles

Perhaps the people who get decent answers don't write articles about them?

I imagine people give up silently more often than they write a well syndicated article about it. The actual adoption and efficiencies we see in enterprises will be the most verifiable data on if LLMs are generally useful in practice. Everything so far is just academic pontificating or anecdata from strangers online.

Re: LLMs aren't world models

#140

Post by Mikhail_Edoshin »

2muchcoffeeman wrote: post #133
libraryofbabel wrote: post #22

Earlier quoted context omitted.

I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how hum…

It’s not just on the job learning though. I’m no AI expert, but the fact that you have “prompt engineers” and AI doesn’t know what it doesn’t know, gives me pause. If you ask an expert, they know the bounds of their knowledge and can understand questions asked to them in multiple ways. If they don’t know the answer, they could point to someone who does or just say “we don’t know”. LLMs just lie to you and we call it…

LLM comprehends, but does not understand. It is interesting to see these two qualities separated; so far they were synonyms.
Post reply on HN