Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on autoregressive models

wonderfall.dev

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Some thoughts on autoregressive models

#41

Post by kevmo314 »

Wonderfall wrote: post #26
eldenring wrote: post #5

> The context window can be compared to working memory in humans: it’s fast, efficient but gets rapidly overloaded. Humans manage this limitation by offloading previously learned information into other memory forms, whereas LLMs can only mimic this process superficially at best. This is just silly. Humans forget things all the time! If I want to remember something I write it down. > The nature of hallucination is ver…

> There's not much signal here, just basic facts about LLMs and then leaps to very bold statements. The article wasn't supposed to be informative for people who already know how LLMs work. Like the title said, just wanted to write down some thoughts. > This is just silly. Humans forget things all the time! If I want to remember something I write it down. The opposite was never stated. Human memory is of course select…

> even for tasks that seem trivial to humans, reasoning LLMs will make a lot of mistakes before arriving at a plausible-sounding result.

Like a lot of my coworkers analyzing a production bug? I would agree if the statement were that LLMs were underpowered compared to a human brain today but I'm not seeing evidence that humans do reasoning in a way that can't be correctly modeled.

From your article and comments, it sounds like the take is something like "humans don't actually reason autoregressively" which could be true, I don't know enough to know, but sort of like saying physics models aren't really how nature works: ultimately LLMs are executable models of the world, it's even in the name.

Re: Some thoughts on autoregressive models

#42

Post by naasking »

dartos wrote: post #15
fergal_reid wrote: post #7

Similar arguments to LeCun. People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could". Large transformers can clearly learn very com…

> while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". Not saying I disagree with your premise that errors can’t be corrected by using more and more tokens, but this argument is weird to me. The model isn’t intentionally generating text. The kinds of “oh let me try a different approach” lines I see are often followed by the same approach just taken. I would…

> The model isn’t intentionally generating text.

What's the mechanistic model of "intention" that you're using to claim that there is no intention in the model's operation?

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

Generating text is the trace of an internal process in an LLM.

Re: Some thoughts on autoregressive models

#43

Post by Wonderfall »

kevmo314 wrote: post #41
Wonderfall wrote: post #26

Earlier quoted context omitted.

> There's not much signal here, just basic facts about LLMs and then leaps to very bold statements. The article wasn't supposed to be informative for people who already know how LLMs work. Like the title said, just wanted to write down some thoughts. > This is just silly. Humans forget things all the time! If I want to remember something I write it down. The opposite was never stated. Human memory is of course select…

> even for tasks that seem trivial to humans, reasoning LLMs will make a lot of mistakes before arriving at a plausible-sounding result. Like a lot of my coworkers analyzing a production bug? I would agree if the statement were that LLMs were underpowered compared to a human brain today but I'm not seeing evidence that humans do reasoning in a way that can't be correctly modeled. From your article and comments, it so…

> From your article and comments, it sounds like the take is something like "humans don't actually reason autoregressively" which could be true, I don't know enough to know, but sort of like saying physics models aren't really how nature works: ultimately LLMs are executable models of the world, it's even in the name.

The conclusion states "Language and thought are not purely autoregressive in humans".

Which doesn't mean humans don't have autoregressive components in their thinking. At least that's my opinion. I don't make this bold statement and I don't know enough to know, too.

> Like a lot of my coworkers analyzing a production bug? I would agree if the statement were that LLMs were underpowered compared to a human brain today

Clearly not in the same way and that was what I was trying to explain with regards to the hallucination issue too. Humans are also learning from proofs, can apply frameworks, etc. there's no denying that. But the internal process of an LLM remains pattern matching and sequential prediction whereas there's more to the human's thinking process.

LLMs are underpowered in some aspects that can't be replicated with autoregressive modeling, but are already stronger in other aspects. That is what I think.

> but I'm not seeing evidence that humans do reasoning in a way that can't be correctly modeled.

Me neither, this is not what my stance is, and I'm actually optimistic about it. I just don't think we should be satisfied with only autoregressive modeling if the ambition is to reach or comprehend human-level intelligence.

Re: Some thoughts on autoregressive models

#44

Post by nyrikki »

TeMPOraL wrote: post #18
dartos wrote: post #15

Earlier quoted context omitted.

> while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". Not saying I disagree with your premise that errors can’t be corrected by using more and more tokens, but this argument is weird to me. The model isn’t intentionally generating text. The kinds of “oh let me try a different approach” lines I see are often followed by the same approach just taken. I would…

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process. What does it represent then? What are all these billion weights for? It's not a bag full of NULLs that just pulls next words from a look-up table. Obviously there is some kind of internal process. Also I don't get why people ignore the temporal aspect. Humans too generat…

Subbarao Kambhampati, who seems to only use X is a good resource. He points out how the CoT text is not of semantic importantce.

This work from his team shows how few 'reasoning' traces are valid.

https://atharva.gundawar.com/searchformer_response_analysis....

This paper shows how the scratch space gets transformers to PTIME from TC0 without it.

https://arxiv.org/abs/2502.02393

OpenAI may be able to do more in the long term because they don't show the and can spend more of that scratch space on improving answers vs appeasing users, but time will show.

Remember that probabilistic checkable proofs show how random data can improve computation.

The AI field has always had a problem with wishful mnomics.

But it is probably not a binary choice, if we could get the scratch space to reliably simulate Dykstra' shunting and convert to postfix as an example, that would be great.

Re: Some thoughts on autoregressive models

#45

Post by _heimdall »

_0ffh wrote: post #35
_heimdall wrote: post #32

Earlier quoted context omitted.

The text represents a prediction of how a human may respond, one word(ish) at a time, that's it. With "reasoning" models, the reasoning layer is basically another LLM instructed to specifically predict how a human may respond to the underlying LLM's answer, fake prompt engineering if you will. There of course is some kind of internal process, but we can't prove any kind of reasoning. We ask a question, the main LLM r…

Please don't confuse people with wrong information, the reasoning part in reasoning models is the exact same LLM that produces the final answer. For example o1 uses special "thinking" tokens to demarcate between reasoning and answer sections of it's output.

Sure, that's a great clarification though maybr a bit of an implementation detail in this context.

Functionally my argument stands in this context - just because we can see one stream of LLM responses responding to the primary response stream says nothing of reasoning or what is going on internally in the reasoning layer.

Re: Some thoughts on autoregressive models

#46

Post by dartos »

_heimdall wrote: post #45
_0ffh wrote: post #35

Earlier quoted context omitted.

Please don't confuse people with wrong information, the reasoning part in reasoning models is the exact same LLM that produces the final answer. For example o1 uses special "thinking" tokens to demarcate between reasoning and answer sections of it's output.

Sure, that's a great clarification though maybr a bit of an implementation detail in this context. Functionally my argument stands in this context - just because we can see one stream of LLM responses responding to the primary response stream says nothing of reasoning or what is going on internally in the reasoning layer.

> what is going on internally in the reasoning layer.

We literally know exactly what is going on with every layer.

It’s well defined. There are mathematical proofs for everything.

Moreover it’s all machine instructions which can be observed.

The emergent properties we see in LLMs are surprising and impressive, but not magic. Internally what is happening is a bunch of matrix multiplications.

There’s no internal thought or process or anything like that.

It’s all “just” math.

To assume anything else is personification bias.

To look at LLMs outputting text and a human writing text and think “oh these two things must be working in the same way” is just… not a very critical line of thought.

Re: Some thoughts on autoregressive models

#47

Post by Tostino »

dartos wrote: post #33
TeMPOraL wrote: post #18

Earlier quoted context omitted.

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process. What does it represent then? What are all these billion weights for? It's not a bag full of NULLs that just pulls next words from a look-up table. Obviously there is some kind of internal process. Also I don't get why people ignore the temporal aspect. Humans too generat…

> Humans too generate thoughts in sequence, You don’t know this. I don’t feel like I generate thoughts in sequence, for me it feels hierarchical. > can't arbitrarily mutate what came before Uhh… what? Do you remember your memories as a child? Or what you ate for breakfast 3 weeks ago? Have you ever misremembered an event or half remembered a solution to a problem? The information in human minds are entirely mutable.…

Not OP.

> Do you remember your memories as a child? Or what you ate for breakfast 3 weeks ago?

For me, this seems like conjuring up and thinking about a childhood event is like putting what came out of my nebulous 'memory' fresh into context at the point in time you are thinking about it, along with whatever thoughts I had about it (how embarrassed I was, how I felt proud because of X, etc). As that context fades into the past, some of those thoughts may get mixed back into that region of my 'memory' associated with that event.

Re: Some thoughts on autoregressive models

#48

Post by aithrowawaycomm »

> You can say LLMs are fundamentally dumb because of their inherent linearity. Are they? Isn’t language by itself linear (more precisely, the presentation of it)?

Any linearity (or at least partial ordering) of intelligence comes from time and causality, not language - in fact the linearity of language is a limitation human cognition struggles to fight against.

I think this is where "chimpanzees are intelligent" comes to the rescue - AI has a nasty habit of focusing too much on humans. It is vacuous to think that chimpanzee intelligence can be reduced to a linear sequence of oohs-and-aahs, although I suspect a transformer trained on thousands of hours of chimp vocalizations could keep a real chimp busy for a long time. Ape cognition is much deeper and more mysterious: imperfect "axioms" and "algorithms" about space, time, numbers, object-ness, identifying other intelligences, etc, seem to be somehow built-in, and all apes seem to share deep cognitive tools like self-reflection, estimating the cognitive complexity of a task, robust quantitative reasoning, and so on. Nor does it really make sense to hand-wave about "evolutionary training data" - there are stark micro- and macro-architectural differences between primate brains and squirrel brains. Not to mention that all species have the exact same amount of data - if it was just about millions of years, why are bees and octopi uniquely intelligent among invertebrates? Why aren't there any chimpanzee-level squirrels? Rather than twisting into knots about "high quality evolutionary data," it makes a lot more sense to point towards evolution pressuring the development of different brain architectures with stronger cognitive abilities. (Especially considering how rapidly modern human intelligence seems to evolved - much more easily explained by sudden favorable mutations vs stumbling into an East African data treasure trove.)

Human intelligence uses these "algorithms" + the more modern tool of language to reason about the world. I believe any AI system which starts with language and sensory input[1], then hopes to get causality/etc via Big Data is doomed to failure: it might be an exceptionally useful text generator/processor but there will be infinite families of text-based problems that toddlers can solve but the AI cannot.

[1] I also think sight-without-touch is doomed to failure, especially with video generation, but that's a different discussion. And AIs can somewhat cheat "touch" if they train extensively on a good video game engine (I see RDR2 is used a lot).

Re: Some thoughts on autoregressive models

#49

Post by hansvm »

> By design, AR models lack planning and reasoning capabilities. If you generate one word at a time, you don’t really have a general idea of where you’re heading.

I have one minor quibble here, which is that the limitation described isn't a criticism of AR models (whose outputs are only "backward-looking" for their inputs), but just a subset of AR models in popular use. An AR model is fully capable of generating a large state space and doing many computations (even doing many full-connected diffusion steps) before generating the first output token.

That quibble wouldn't be worth mentioning unless AR models had some sort of advantage, but they do, and it's incredibly important. AR factorization of the conditional probabilities allows you to additively consider the loss contribution from each output token -- you can blindly shove whatever data you want into the thing, add up all the errors, and backpropagate, all while guaranteeing that the distribution you're learning is the same distribution from your training data.

If you're not careful, via some mechanism (like AR), the distribution you learn will have almost nothing to do with the distribution you're training on -- a common failure mode being a tendancy to predict "average-looking" sub-tiles in a composite image and only predict images which can be comprised out of those smaller, averge-looking sub-tiles. Imagine (as an example, with low enough model capacity), you had a model generating people and everyone was vaguely 5'10", ambiguously gendered, and a bit tan, contrasted with that same model trained using AR where you'd expect the outputs to be bad in other ways if you had insufficient capacity but to at least have a mix of colors, heights, and genders. Increasing capacity can help, but why bother when something like AR solves it by definition?

Re: Some thoughts on autoregressive models

#50

Post by _heimdall »

dartos wrote: post #46
_heimdall wrote: post #45

Earlier quoted context omitted.

Sure, that's a great clarification though maybr a bit of an implementation detail in this context. Functionally my argument stands in this context - just because we can see one stream of LLM responses responding to the primary response stream says nothing of reasoning or what is going on internally in the reasoning layer.

> what is going on internally in the reasoning layer. We literally know exactly what is going on with every layer. It’s well defined. There are mathematical proofs for everything. Moreover it’s all machine instructions which can be observed. The emergent properties we see in LLMs are surprising and impressive, but not magic. Internally what is happening is a bunch of matrix multiplications. There’s no internal though…

> We literally know exactly what is going on with every layer.

Unless I missed a huge break in the observability problem, this isn't correct.

We know exactly how every layer is designed and we know how we functionally expect that to work. We don't know what actually happens in the model at time of inference.

I.e. we know what pieces were used to build the thing but when we actually use it its a black box - we only know inputs and outputs.

Post reply on HN