Self-reasoning tokens: teaching models to think ahead
reasoning-tokens.ghost.io
Self-reasoning tokens: teaching models to think ahead
1–10 of 31 posts
Re: Self-reasoning tokens: teaching models to think ahead
#2Ok so my understanding: you can have the network generate a token that can be used as input to future token generation along with each output token it generates
These are called reasoning tokens
Initial results with gpt2 are promising
You can generalize this to let the network decide when to generate reasoning tokens (I'm unclear on how). There were also multiple lines in the loss graph with reasoning tokens that I don't quite understand (what's reasoning 1 vs 3? Is it the ratio of reasoning tokens? Something else?)
Re: Self-reasoning tokens: teaching models to think ahead
#3Ok so my understanding: you can have the network generate a token that can be used as input to future token generation along with each output token it generates These are called reasoning tokens Initial results with gpt2 are promising You can generalize this to let the network decide when to generate reasoning tokens (I'm unclear on how). There were also multiple lines in the loss graph with reasoning tokens that I d…
Reasoning 1 vs. 3 is the number of reasoning tokens between each "text" token. The 1 reasoning token is exactly what you see in the picture explanation in the article.
The generalization comes from making the network predict a and end the sequence only when it predicts a . The training dataset for the upcoming experiment contains examples like: """ Q: What is 3+2? A: 3 + 2 is equal to ... 5 """
Re: Self-reasoning tokens: teaching models to think ahead
#4I have definately and frustratingly seen GPT3.5-Turbo do a bunch of anticipation in the outputs.
Basically it will create pre-conditions so that the final output aligns to some bias. In my specific case it was the bias to provide an answer to a question. This is noticable sometimes in chain of thought intermediate outputs. I ended up having to create some space between the entangled decisions in the chain of thought output.
Re: Self-reasoning tokens: teaching models to think ahead
#5I’ve tried similar experiments before by asking the LLM to generate “internal” and “external” dialog, which I think is sort of the same idea at a higher level—-and might be preferable because it would allow for easy introspection vs a new set of tokens? I’m not enough of an expert to understand whether this proposal is intended more for training or inference.
Re: Self-reasoning tokens: teaching models to think ahead
#6I’ve tried similar experiments before by asking the LLM to generate “internal” and “external” dialog, which I think is sort of the same idea at a higher level—-and might be preferable because it would allow for easy introspection vs a new set of tokens? I’m not enough of an expert to understand whether this proposal is intended more for training or inference.
The main advantage of using a new and constant token for reasoning is that, while we would pay the full price during training, in the inference phase, we could do most, if not all, the "reasoning" in one shot, without having to feed one generation token at a time.
Re: Self-reasoning tokens: teaching models to think ahead
#7I’ve tried similar experiments before by asking the LLM to generate “internal” and “external” dialog, which I think is sort of the same idea at a higher level—-and might be preferable because it would allow for easy introspection vs a new set of tokens? I’m not enough of an expert to understand whether this proposal is intended more for training or inference.
This method is for training. They are using a stop-gradient to 'shield' some tokens from contributing to prediction of the immediate next token, and thus producing a stream of tokens that are only used for longer term prediction.
This is a bit more low level than the usual prompt engineering approaches, and to my mind, a bit more promising. There's more easily measurable results, and I've seen other context where a well placed stop-gradient does wonders...
Re: Self-reasoning tokens: teaching models to think ahead
#8I’ve tried similar experiments before by asking the LLM to generate “internal” and “external” dialog, which I think is sort of the same idea at a higher level—-and might be preferable because it would allow for easy introspection vs a new set of tokens? I’m not enough of an expert to understand whether this proposal is intended more for training or inference.
The main advantage of using a new and constant token for reasoning is that, while we would pay the full price during training, in the inference phase, we could do most, if not all, the "reasoning" in one shot, without having to feed one generation token at a time.
Cool!
Re: Self-reasoning tokens: teaching models to think ahead
#9For the existing models is beam-search like methods hopeless due to combinatorial explosion? Are there no smart ways to improve it? Evaluating multiple futures will be slow but if it means that the model can give vastly better output, it might be a worthwhile trade-off in some cases. I feel like our standard way of sampling the output of the LLMs is a bit too simplistic and my hunch is that it should be possible to get a lot more out of them even if it means losing speed.
Re: Self-reasoning tokens: teaching models to think ahead
#10"The second token, however, duplicates the input of the first one and does not receive a gradient "answer" from the very next token, only from future tokens; ..."
This formulation doesn't make a lot of sense to me.
I get the motivation here but what you're trying to implement is a working memory.
Because transformers have perfect retrospective memory within their context window any generation which can be done directly from input tokens will be.
At any given point a model might want to write to a working memory, but that does not imply that the next non-working-memory-step will supply useful information to better write to working memory in the future. The model also has to be able to decide when to compare the work done in working memory to the next token.
By allowing the model to both exempt output from gradient updates and opt back in to gradient updates, you create a meta-learning loop that could be quite flexible.