Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-reasoning tokens: teaching models to think ahead

reasoning-tokens.ghost.io

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Self-reasoning tokens: teaching models to think ahead

#11

Post by PeterisP »

XenophileJKO wrote: post #4

I have definately and frustratingly seen GPT3.5-Turbo do a bunch of anticipation in the outputs. Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought…

> In my specific case it was the bias to provide an answer to a question

That seems to be a reasonably expected result of the "instruction post-training" finetuning with RLHF or otherwise. If for some reason you don't want this behavior, you can avoid this by using a model version that just has the core language modeling without that finetuning, e.g. the llama models have such a version available.

Re: Self-reasoning tokens: teaching models to think ahead

#13

Post by pizza »

wrsh07 wrote: post #2

Ok so my understanding: you can have the network generate a token that can be used as input to future token generation along with each output token it generates These are called reasoning tokens Initial results with gpt2 are promising You can generalize this to let the network decide when to generate reasoning tokens (I'm unclear on how). There were also multiple lines in the loss graph with reasoning tokens that I d…

I'm just speculating here since I don't know what or where the code is but since inference is still autoregressive;

given [a b c] sample [d]

distribution of [d] could be over [reasoning token] | [vocab token]

then at next step you have

[a b c d] and each has an embedding vector associated

so when you go to sample [e] it's a function of [a b c d]

Re: Self-reasoning tokens: teaching models to think ahead

#14

Post by HarHarVeryFunny »

earslap wrote: post #9

For the existing models is beam-search like methods hopeless due to combinatorial explosion? Are there no smart ways to improve it? Evaluating multiple futures will be slow but if it means that the model can give vastly better output, it might be a worthwhile trade-off in some cases. I feel like our standard way of sampling the output of the LLMs is a bit too simplistic and my hunch is that it should be possible to g…

People are considering that sort of beam-search approach - this is what they call "tree of thoughts" - generate a branching tree of alternate continuations, then pick the best one based on some criteria.

This doesn't seem an ideal approach though, since it amounts to generating a bunch of shallow responses and picking the best, rather than the preferred thinking more deeply before generating. It's not the same as a computer chess program considering N-moves ahead where you are guaranteed that one of those move sequences really is the best one (as long as you don't accidentally prune it out). In contrast, if you generate all possible "shallow" N-token responses (bunch of monkeys gibbering), there is no guarantee any of those will be the high quality response you are hoping for.

Really planning ahead - reasoning deeply before speaking - would seem harder to implement though, since it'd involve applying a variable number of reasoning steps (maybe looping), then determining when to stop. This also seems different from the proposed insertion of "reasoning tokens" since those are shallow reasoning steps (normal single pass through transformer's layers), when it seems what is really needed is more depth of reasoning ("more layers"), perhaps coupled with some working memory/tokens. Both schemes (more tokens vs more depth) are also related to the wish to use a variable amount of compute for different tasks/inputs - less compute for simple tasks, more for hard ones.

Re: Self-reasoning tokens: teaching models to think ahead

#15

Post by alt0_ »

XenophileJKO wrote: post #4

I have definately and frustratingly seen GPT3.5-Turbo do a bunch of anticipation in the outputs. Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought…

> definately

relevant xkcd: https://xkcd.com/2871/

Re: Self-reasoning tokens: teaching models to think ahead

#16

Post by pizza »

exploringBytes wrote: post #12

First association was to extend the modality of text tokens to concept tokens which could be (logical) relationships. Are you aware of similar works?

Not exactly the same game but you might be interested in Mathematical Structure of Syntactic Merge, Marcolli, Chomsky, Berwick (2023).

When we speak we give a string. When we think we don't have to use a string. But we do have to have a functionality to map something that has no single ordering to something that has an ordering (externalization) - a sentence. And vice versa we have a functionality to turn strings into things without a specific ordering (internalization) - thoughts.

Re: Self-reasoning tokens: teaching models to think ahead

#17

Post by sdwr »

wantsanagent wrote: post #10

"The second token, however, duplicates the input of the first one and does not receive a gradient "answer" from the very next token, only from future tokens; ..." This formulation doesn't make a lot of sense to me. I get the motivation here but what you're trying to implement is a working memory. Because transformers have perfect retrospective memory within their context window any generation which can be done direct…

As I understand it, this isn't trying to implement actual memory in the form of a cache, but instead some kind of wishy-washy memory-lite.

I'm talking out of my ass here, but I feel like real memory shouldn't be that hard to implement on top of chatGPT. Just run it twice per query, the first time as an internal query that fetches from a memory store.

The budgeting part would be interesting. How many tokens of the main query do you want to fill with memories? And it wouldn't be able to meta learn how to use the system better, you'd have to update the prompt

Re: Self-reasoning tokens: teaching models to think ahead

#18

Post by XenophileJKO »

alt0_ wrote: post #15
XenophileJKO wrote: post #4

I have definately and frustratingly seen GPT3.5-Turbo do a bunch of anticipation in the outputs. Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought…

> definately relevant xkcd: https://xkcd.com/2871/

If I wanted it spelled correctly I would have run it through the LLM.

Re: Self-reasoning tokens: teaching models to think ahead

#19

Post by XenophileJKO »

PeterisP wrote: post #11
XenophileJKO wrote: post #4

I have definately and frustratingly seen GPT3.5-Turbo do a bunch of anticipation in the outputs. Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought…

> In my specific case it was the bias to provide an answer to a question That seems to be a reasonably expected result of the "instruction post-training" finetuning with RLHF or otherwise. If for some reason you don't want this behavior, you can avoid this by using a model version that just has the core language modeling without that finetuning, e.g. the llama models have such a version available.

Well in this specific case, the logic I was asking the model to do was. (Highly paraphrased..)

1. Inventory the retrieved items.

2. Determine their relevance.

3. Pick the most relevant or if none of the retrieved items is relevant return an alternative message.

What the model will do is add new items into (1) if none of the retrieved items are relevant. If you add some steps between 1 and 2.. it stops doing that.

Re: Self-reasoning tokens: teaching models to think ahead

#20

Post by lucidrains »

sdenton4 wrote: post #7
jacobsimon wrote: post #5

I’ve tried similar experiments before by asking the LLM to generate “internal” and “external” dialog, which I think is sort of the same idea at a higher level—-and might be preferable because it would allow for easy introspection vs a new set of tokens? I’m not enough of an expert to understand whether this proposal is intended more for training or inference.

This method is for training. They are using a stop-gradient to 'shield' some tokens from contributing to prediction of the immediate next token, and thus producing a stream of tokens that are only used for longer term prediction. This is a bit more low level than the usual prompt engineering approaches, and to my mind, a bit more promising. There's more easily measurable results, and I've seen other context where a w…

yes, it is a stop gradient mask on the attention matrix, iiuc. worth trying
Post reply on HN