Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generative AI's failure to induce robust models of the world

garymarcus.substack.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: Generative AI's failure to induce robust models of the world

#31

Post by Lerc »

voidhorse wrote: post #28
Lerc wrote: post #25

Earlier quoted context omitted.

That still seems unfalsifiable. If it fails one instance the claim is that the failure is representative of things outside the training set. If it succeeds the claim is that it is in the training set. Without a definitive way to say something is not in the training set (a likely impossible task) the measure of success or failure is the only indicator of the purported reason reason for the success or failure. Given mo…

I do think there are cases which, in controlled environments, there is some degree of knowledge as to what is in the training set. I also don't thin it's as impossible as you assume. If you really wanted to ensure this with certainty just use the natural numbers to parameterize an aspect of a general problem. Assume there are N foo problems in the training set, then there is always a case N+1 parameter not in the tra…

Well there are models that can sum two many-digit numbers. They certainly have not been trained on every pair of integers up to that level. That either makes the claim they can't do things that they haven't seen trivially false, or the criteria for counting something as being in the training data includes a degree of inference.

What happens when someone makes a claim that they have gotten a model to do something not in the training data and another person claims it must be encoded in the training data in some form. It seems like an impasse.

Re: Generative AI's failure to induce robust models of the world

#32

Post by Animats »

That LLMs are a black box and that LLMs lack an underlying model are both true, but orthogonal. It's possible to have a black box system which has an underlying model. That's true of many statistical prediction methods. Early attempts at machine learning were a white box with no underlying model. This is true of most curve-fitting. The AI version was where you're trying to divide a high-dimensional space with a cutting plane to create a classifier. You can tell where the separating plane is, but not why.

The lack of a world model is a very real limitation in some problem spaces, starting with arithmetic. But this argument is unconvincing.

Re: Generative AI's failure to induce robust models of the world

#33

Post by comp_throw7 »

Animats wrote: post #32

That LLMs are a black box and that LLMs lack an underlying model are both true, but orthogonal. It's possible to have a black box system which has an underlying model. That's true of many statistical prediction methods. Early attempts at machine learning were a white box with no underlying model. This is true of most curve-fitting. The AI version was where you're trying to divide a high-dimensional space with a cutti…

> LLMs lack an underlying model

Obviously false for any useful sense by which you might operationalize "world model". But agree re: being a black box and having a world model being orthogonal.

Re: Generative AI's failure to induce robust models of the world

#34

Post by cageface »

voidhorse wrote: post #10

The whole thing is silly. Look, we know that LLMs are just really good word predictors. Any argument that they are thinking is essentially predicated on marketing materials that embrace anthropomorphic metaphors to an extreme degree. Is it possible that reason could emerge as the byproduct of being really good at predicting words? Maybe, but this depends on the antecedent claim that much if not all of reason is stric…

but this depends on the antecedent claim that much if not all of reason is strictly representational and strictly linguistic.

Most of these newer models are multi-modal, so tokens aren't necessary linguistic.

Re: Generative AI's failure to induce robust models of the world

#35

Post by comp_throw7 »

voidhorse wrote: post #10

The whole thing is silly. Look, we know that LLMs are just really good word predictors. Any argument that they are thinking is essentially predicated on marketing materials that embrace anthropomorphic metaphors to an extreme degree. Is it possible that reason could emerge as the byproduct of being really good at predicting words? Maybe, but this depends on the antecedent claim that much if not all of reason is stric…

What use of the word "reasoning" are you trying to claim that current language models knowably fail to qualify for, except that it wasn't done by a human?

Re: Generative AI's failure to induce robust models of the world

#36

Post by bitmasher9 »

SubiculumCode wrote: post #9

I definitely would be okay if we hit an AI winter; our culture and world cannot adapt fast enough for the change we are experiencing. In the meantime, the current level of AI is just good enough to make us more productive, but not so good as to make us irrelevant.

I think negative feedback loops of AIs trained on AI generated data might lead to a position where AI quality peaks and slides backwards.

Re: Generative AI's failure to induce robust models of the world

#37

Post by NitpickLawyer »

nmadden wrote: post #30
NitpickLawyer wrote: post #19

Earlier quoted context omitted.

> If nothing else, his perspective is a counterbalance to what is clearly an extreme hype machine that is doing its utmost to force adoption through overpromising, false advertising, etc. These are bad things even if the tech does actually have some useful applications. In other words, overhyped in the short term, underhyped in the long term. Where short and long term are extremely volatile. Take programming as an ex…

The improvements in programming are largely due to the adoption of “agentic” architectures. This is really a hybrid neural-symbolic approach: the symbolic part being the interpreter/compiler. Effectively the LLM still produces an almost-correct-but-wrong program and then the compiler “fact-checks” it and then the LLM basically local-searches its way from there to something that passes the compiler. (If you want to be…

> The improvements in programming are largely due to the adoption of “agentic” architectures.

Yes, I agree. But it's not just the cradles, it's cradles + training on traces produced with those cradles. You can test this very easily with running old models w/ new cradles. They don't perform well at all. (one of the first things I did when guidance, a guided generation framework, launched ~2 years ago was to test code - compile - edit loops. There were signs of it working, but nothing compared to what we see today. That had to be trained into the models.)

> will not come from better LLM models (which have not really improved much), but from better integration of more advanced compilers.

Strong disagree. They have to work together. This is basically why RL is gaining a lot of traction in this space.

Also disagree on llms not improving much. Whatever they did with gemini 2.5 feels like gpt3-4 to me. The context updates are huge. This is the first model that can take 100k tokens and still work after that. They're doing something right to be able to support such large contexts with such good performance. I'd be surprised if gemini 2.5 is just gemini 1 + more data. Extremely surprised. There have to be architecture changes and improvements somewhere in there.

Re: Generative AI's failure to induce robust models of the world

#38

Post by sgt101 »

bitmasher9 wrote: post #36
SubiculumCode wrote: post #9

I definitely would be okay if we hit an AI winter; our culture and world cannot adapt fast enough for the change we are experiencing. In the meantime, the current level of AI is just good enough to make us more productive, but not so good as to make us irrelevant.

I think negative feedback loops of AIs trained on AI generated data might lead to a position where AI quality peaks and slides backwards.

Thank goodness we have version control systems then.

Re: Generative AI's failure to induce robust models of the world

#39

Post by dist-epoch »

The article links to a tweet about jail-braking Claude to provide a recipe for Sarin gas production: https://x.com/argleave/status/1926138376509440433

But some words are redacted. So I've uploaded the picture to Gemini and asked it what the redacted words are, and it told me. Not sure if they are correct, and some are way longer to fit in the redacted black box, but it didn't refuse the request.

Re: Generative AI's failure to induce robust models of the world

#40

Post by fmbb »

viraptor wrote: post #20
deadbabe wrote: post #4

Earlier quoted context omitted.

Even if you find a pheromone trail, it doesn’t tell you what direction is home, or what path to take at branching paths. You need dead reckoning. The trail just helps you reduce the complexity of what you have to remember.

The lack of information in ant trails (beyond "it exists here") leads to death spirals https://en.m.wikipedia.org/wiki/Ant_mill

The very first sentence of the article you linked says this happens because they lose the pheromone track.
Post reply on HN