Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on autoregressive models

wonderfall.dev

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Some thoughts on autoregressive models

#11

Post by yorwba »

fergal_reid wrote: post #7

Similar arguments to LeCun. People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could". Large transformers can clearly learn very com…

As the number of self-corrections increases, it also increases the likelihood that it will say "oh that's not right, let me try a different approach" after finding the correct solution. Then you can get into a second-guessing loop that never arrives at the correct answer.

If the self-check is more reliable than the solution-generating process, that's still an improvement, but as long as the model makes small errors when correcting itself, those errors will still accumulate. On the other hand, if you can have a reliable external system do the checking, you can actually guarantee correctness.

Re: Some thoughts on autoregressive models

#12

Post by toxik »

ziofill wrote: post #9

Is anyone aware of a formalization of the idea that to get “symbols” out of fuzzy probability distributions one needs distributions whose value goes exactly to zero over some regions of the domain? I.e. Gaussian mixtures won’t cut it. And they will need very high Fourier frequencies. I have the gut feeling that until a model allows for a small probability that 2x3 is 7, there will always be hallucinations. Probabilit…

We do clamp probabilities to zero. Look into top-p sampling or nucleus sampling.

Re: Some thoughts on autoregressive models

#14

Post by mxwsn »

> But what is the original purpose of AI research? I will speak for myself here, but I know many other AI researchers will say the same: the ultimate goal is to understand how humans think. And we think the best (or the funniest) way to understand how humans think is to try to recreate it.

Eh. To riff on Dijkstra, this is like submarine engineers saying their ultimate goal is to understand how fish swim.

Re: Some thoughts on autoregressive models

#15

Post by dartos »

fergal_reid wrote: post #7

Similar arguments to LeCun. People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could". Large transformers can clearly learn very com…

> while we literally watch reasoning models say things like "oh that's not right, let me try a different approach".

Not saying I disagree with your premise that errors can’t be corrected by using more and more tokens, but this argument is weird to me.

The model isn’t intentionally generating text. The kinds of “oh let me try a different approach” lines I see are often followed by the same approach just taken. I wouldn’t say most of the time, but often enough that I notice.

Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

Re: Some thoughts on autoregressive models

#16

Post by solveit »

yorwba wrote: post #11
fergal_reid wrote: post #7

Similar arguments to LeCun. People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could". Large transformers can clearly learn very com…

As the number of self-corrections increases, it also increases the likelihood that it will say "oh that's not right, let me try a different approach" after finding the correct solution . Then you can get into a second-guessing loop that never arrives at the correct answer. If the self-check is more reliable than the solution-generating process, that's still an improvement, but as long as the model makes small errors…

Error correction is possible even if the error correction is itself noisy. The error does not need to accumulate, it can be made as small as you like at the cost of some efficiency. This is not a new problem, the relevant theorems are incredibly robust and have been known for decades.

Re: Some thoughts on autoregressive models

#17

Post by raylad »

eldenring wrote: post #5

> The context window can be compared to working memory in humans: it’s fast, efficient but gets rapidly overloaded. Humans manage this limitation by offloading previously learned information into other memory forms, whereas LLMs can only mimic this process superficially at best. This is just silly. Humans forget things all the time! If I want to remember something I write it down. > The nature of hallucination is ver…

Do you think in words when you do a 3 x 2 digit multiplication?

I do it all in images and I think many other people do too.

Re: Some thoughts on autoregressive models

#18

Post by TeMPOraL »

dartos wrote: post #15
fergal_reid wrote: post #7

Similar arguments to LeCun. People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could". Large transformers can clearly learn very com…

> while we literally watch reasoning models say things like "oh that's not right, let me try a different approach". Not saying I disagree with your premise that errors can’t be corrected by using more and more tokens, but this argument is weird to me. The model isn’t intentionally generating text. The kinds of “oh let me try a different approach” lines I see are often followed by the same approach just taken. I would…

> Just because a model generates text doesn’t mean that the text actually represents anything at all, let alone a reflection of an internal process.

What does it represent then? What are all these billion weights for? It's not a bag full of NULLs that just pulls next words from a look-up table. Obviously there is some kind of internal process.

Also I don't get why people ignore the temporal aspect. Humans too generate thoughts in sequence, and can't arbitrarily mutate what came before. Time and memory is what forces sequential order - we too just keep piling on more thoughts to correct previous thoughts while they are still in working memory (context).

Re: Some thoughts on autoregressive models

#19

Post by yorwba »

solveit wrote: post #16
yorwba wrote: post #11

Earlier quoted context omitted.

As the number of self-corrections increases, it also increases the likelihood that it will say "oh that's not right, let me try a different approach" after finding the correct solution . Then you can get into a second-guessing loop that never arrives at the correct answer. If the self-check is more reliable than the solution-generating process, that's still an improvement, but as long as the model makes small errors…

Error correction is possible even if the error correction is itself noisy. The error does not need to accumulate, it can be made as small as you like at the cost of some efficiency. This is not a new problem, the relevant theorems are incredibly robust and have been known for decades.

Can you link me to a proof demonstrating that the error can be made arbitrarily small? (Or at least a precise statement of the theorem you have in mind.) I would think that if the last step of error correction turns a correct intermediate result into an incorrect final result with probability p, that puts a lower bound of p on the overall error rate.

Re: Some thoughts on autoregressive models

#20

Post by TeMPOraL »

ziofill wrote: post #9

Is anyone aware of a formalization of the idea that to get “symbols” out of fuzzy probability distributions one needs distributions whose value goes exactly to zero over some regions of the domain? I.e. Gaussian mixtures won’t cut it. And they will need very high Fourier frequencies. I have the gut feeling that until a model allows for a small probability that 2x3 is 7, there will always be hallucinations. Probabilit…

Symbolic behavior is artificial and not how humans think either. 0 is not a probability (neither is 1) - a value of 0 or 1 basically breaks calculations by dragging everything along to the limit, the same way infinity does, or 0 in the denominator (in fact, that's what 1 and 0 translate to if you switch to logprobs or other equivalent ways to calculate probabilities).

Consider: if you clamp the probability distribution of answers to 2x3, so that it's 0 everywhere else and 1 at 6, you're basically saying that it is fundamentally impossible for you to misunderstand the question, or make mistake in the answer, or that you're dreaming, or hallucinating, or that you've momentarily forgotten that the question was preceded by "In base 4, what is ", or any number of other things that absolutely are possible, even if highly unlikely, in the real world.

Post reply on HN