Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generative AI's failure to induce robust models of the world

garymarcus.substack.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Generative AI's failure to induce robust models of the world

#61

Post by daveguy »

energy123 wrote: post #58
csande17 wrote: post #52

Earlier quoted context omitted.

There's a pretty big difference between AlphaZero and a "generative AI" program: AlphaZero has access to an oracle that can tell it whether it's making valid moves and winning games. By comparison, getting accurate feedback on whether facts are correct in a piece of text (for example) is much more difficult and expensive. At least, presumably that's why AI companies publish staged demo videos where the AI still makes…

Automatic verification (oracle) is being used today to create synthetic data for LLMs. I don't see it as a big difference versus AlphaZero. While there's no way to ensure that a single synthetic reasoning trace is correct, as long as it leads to the correct answer according to the verifier, the law of large numbers should take care of that. The problem is that it's difficult to create verifiers for many things we car…

Transfer learning is one of the biggest unsolved problems in AI. And we are nowhere near solving it or even understanding how to go about it from an algorithmic perspective. We will definitely see collapse of the current hype train before we understand and employ effective transfer learning.

Re: Generative AI's failure to induce robust models of the world

#62

Post by daveguy »

Lerc wrote: post #18
squirrel wrote: post #14

Earlier quoted context omitted.

He cites o3 and o4-mini as examples of LLMs that play illegal chess moves.

I don't understand the reasoning behind drawing a conclusion that if something fails a task that requires reasoning implies that thing cannot reason. To use chess as an example. Humans sometimes play illegal moves. That does not mean Humans cannot reason. It is an instance of failing to show proof of reasoning. Not a proof of the inability to reason.

Humans who know how to play chess do not play illegal chess moves. Humans can learn chess in an afternoon and never make an illegal move again. The rules are pretty simple, and they are rules that every LLM has seen dozens of not hundreds of times in their training data. They still play illegal moves because they are not learning anything except how to simulate conversation.

Another algorithmic learning breakthrough, on the order of perceptrons, deep learning, transformers, etc is necessary to get anywhere near AGI.

Re: Generative AI's failure to induce robust models of the world

#63

Post by Jensson »

energy123 wrote: post #29
voidhorse wrote: post #24

Earlier quoted context omitted.

I don't think that's a fair representation of the argument. The argument is not "here's one failure case, therefore they don't reason". The argument is that systematically if you given an LLM problem instances outside training sets in domains with clear structural rules , they will fail to solve them. The argument then goes that they must not have an actual model or understanding of the rules, as they seem to only be…

The lack of rigor and evidence behind the argument is the problem.

It is the side that is arguing that it is reasoning that is lacking rigor and evidence. The side that arguing it isn't is saying you need more rigor and evidence when you claim it is reasoning by pointing out simple cases where it fails.

Re: Generative AI's failure to induce robust models of the world

#64

Post by daveguy »

Lerc wrote: post #59
imtringued wrote: post #48

Earlier quoted context omitted.

Anthropomorphic fallacy. Human fails at task due to not knowing the rules in perfect detail. AI fails at task even though it knows the rules and could easily reproduce them for chess and dozens of chess variants. "Look! The fallibility of humans rubbed off onto the AI, proving that they are more human and AGI than we give them credit to!"

I'm not sure how you consider this to be an anthropomorphic fallacy, the comparison to the situation with a human exists only because people are prepared to stipulate that humans can reason. That does not assume something about AI behaviour to be like a human's. It is showing the same test applied to a human. Your statement that AI knows the rules would be considered anthropomorphising by many, I take it more to mean…

> It is showing the same test applied to a human.

But you misrepresented the test with respect to humans. Humans who know how to play chess don't make illegal moves.

> That said, humans who have written entire books on chess have been known to play illegal moves.

Citation needed. Unless you are talking about stories from when they first learned the rules?

Re: Generative AI's failure to induce robust models of the world

#65

Post by trainerxr50 »

voidhorse wrote: post #10

The whole thing is silly. Look, we know that LLMs are just really good word predictors. Any argument that they are thinking is essentially predicated on marketing materials that embrace anthropomorphic metaphors to an extreme degree. Is it possible that reason could emerge as the byproduct of being really good at predicting words? Maybe, but this depends on the antecedent claim that much if not all of reason is stric…

I think more importantly there is this stupid argument that because the submarine is not swimming it will never be able to "swim" as fast as us.

This is true of course in a pointlessly rhetorical sense.

Completely absurd though once we change "swimming" to the more precise "moving through water".

The solution is not to put arms and legs on the submarine so it can ACTUALLY swim.

It would be quite trivial to make a Gary Marcus style argument that humans still can't fly. We would need much longer and wider arms, much less core body mass, feathers.

Re: Generative AI's failure to induce robust models of the world

#66

Post by daveguy »

NitpickLawyer wrote: post #37
nmadden wrote: post #30

Earlier quoted context omitted.

The improvements in programming are largely due to the adoption of “agentic” architectures. This is really a hybrid neural-symbolic approach: the symbolic part being the interpreter/compiler. Effectively the LLM still produces an almost-correct-but-wrong program and then the compiler “fact-checks” it and then the LLM basically local-searches its way from there to something that passes the compiler. (If you want to be…

> The improvements in programming are largely due to the adoption of “agentic” architectures. Yes, I agree. But it's not just the cradles, it's cradles + training on traces produced with those cradles. You can test this very easily with running old models w/ new cradles. They don't perform well at all. (one of the first things I did when guidance, a guided generation framework, launched ~2 years ago was to test code…

> You can test this very easily with running old models w/ new cradles. They don't perform well at all.

This is because neither the LLMs nor the cradles are intelligent.

> They have to work together.

Exactly. Because they are essentially a single, brittle model. Not a "smart" text generator + a "smart" validation system.

LLMs are an enormous breakthrough in NLP and something like it will be part of an AGI system. But there is no path to AGI without more breakthroughs.

Re: Generative AI's failure to induce robust models of the world

#67

Post by Lerc »

daveguy wrote: post #64
Lerc wrote: post #59

Earlier quoted context omitted.

I'm not sure how you consider this to be an anthropomorphic fallacy, the comparison to the situation with a human exists only because people are prepared to stipulate that humans can reason. That does not assume something about AI behaviour to be like a human's. It is showing the same test applied to a human. Your statement that AI knows the rules would be considered anthropomorphising by many, I take it more to mean…

> It is showing the same test applied to a human. But you misrepresented the test with respect to humans. Humans who know how to play chess don't make illegal moves. > That said, humans who have written entire books on chess have been known to play illegal moves. Citation needed. Unless you are talking about stories from when they first learned the rules?

https://www.chess.com/blog/kranthimanaswi/top-5-illegal-move...

Re: Generative AI's failure to induce robust models of the world

#68

Post by bloaf »

danaris wrote: post #60
seanhunter wrote: post #46

Earlier quoted context omitted.

“LLMs lack an underlying model” is very obviously incorrect. LLMs have an underlying model of semantics as tokens embedded into a high-dimensional vector space. The question is not whether or not they have any model at all, the question is whether the model they indisputably have (which is a model of language in terms of linear algebra) maps onto a model of the external universe (a “world model”) that emerges during…

I agree that it's probably unfalsifiable in the sense of proving it definitively based on something like static analysis of the model itself. But that doesn't mean that we can't, in theory, give the LLM a battery of tests that it should perform well (though not perfectly) on if it has a world model, and poorly (though not fail totally) on if it doesn't. It's inherently a probabilistic system, so testing it in a proba…

I don't think its nearly as cut-and-dry as that. Even if you tried to make tests to differentiate world-model from non-world-model, all you'd end up concluding is:

If the AI has a world model, its world-model doesn't have features that allow it to do what I tested for.

Re: Generative AI's failure to induce robust models of the world

#69

Post by deadbabe »

viraptor wrote: post #20
deadbabe wrote: post #4

Earlier quoted context omitted.

Even if you find a pheromone trail, it doesn’t tell you what direction is home, or what path to take at branching paths. You need dead reckoning. The trail just helps you reduce the complexity of what you have to remember.

The lack of information in ant trails (beyond "it exists here") leads to death spirals https://en.m.wikipedia.org/wiki/Ant_mill

How could they encode some kind of directional information into a trail?

Re: Generative AI's failure to induce robust models of the world

#70

Post by daveguy »

Lerc wrote: post #67
daveguy wrote: post #64

Earlier quoted context omitted.

> It is showing the same test applied to a human. But you misrepresented the test with respect to humans. Humans who know how to play chess don't make illegal moves. > That said, humans who have written entire books on chess have been known to play illegal moves. Citation needed. Unless you are talking about stories from when they first learned the rules?

https://www.chess.com/blog/kranthimanaswi/top-5-illegal-move...

Did you read those? These are the "illegal" moves listed:

5. Mouse slip

4. Forgot to call check

3. Accidentally touched 2 pieces, tried to fix it

2. Forgot to hit the clock button

1. Castle through attacked square

So, the only one of these that was an acual "illegal move" of the sort LLMs make was the castle through attacked square.

LLMs sometimes just move pieces wherever. And that does not happen when humans who know the rules play. Yes, they may mess up en passant or promotion too. But a basic "how a single piece moves" rule is what LLMs f up.

Post reply on HN