Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-reasoning tokens: teaching models to think ahead

reasoning-tokens.ghost.io

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Self-reasoning tokens: teaching models to think ahead

#21

Post by lucidrains »

lucidrains wrote: post #20
sdenton4 wrote: post #7

Earlier quoted context omitted.

This method is for training. They are using a stop-gradient to 'shield' some tokens from contributing to prediction of the immediate next token, and thus producing a stream of tokens that are only used for longer term prediction. This is a bit more low level than the usual prompt engineering approaches, and to my mind, a bit more promising. There's more easily measurable results, and I've seen other context where a w…

yes, it is a stop gradient mask on the attention matrix, iiuc. worth trying

could even try it with a fraction of the attention heads, instead of introducing new tokens

Re: Self-reasoning tokens: teaching models to think ahead

#22

Post by earslap »

HarHarVeryFunny wrote: post #14
earslap wrote: post #9

For the existing models is beam-search like methods hopeless due to combinatorial explosion? Are there no smart ways to improve it? Evaluating multiple futures will be slow but if it means that the model can give vastly better output, it might be a worthwhile trade-off in some cases. I feel like our standard way of sampling the output of the LLMs is a bit too simplistic and my hunch is that it should be possible to g…

People are considering that sort of beam-search approach - this is what they call "tree of thoughts" - generate a branching tree of alternate continuations, then pick the best one based on some criteria. This doesn't seem an ideal approach though, since it amounts to generating a bunch of shallow responses and picking the best, rather than the preferred thinking more deeply before generating. It's not the same as a c…

Ah yes, I totally agree. I was inspecting the method as a stopgap solution (especially because it does not require retraining or any other special tricks) until researchers figure out "planning" in a broader sense. It is very inefficient otherwise, but in the meantime, is just simple sampling with a couple parameters to tune from the output softmax the best we can do? is there no low hanging fruit there?

Re: Self-reasoning tokens: teaching models to think ahead

#24

Post by wrsh07 »

fesens wrote: post #3
wrsh07 wrote: post #2

Ok so my understanding: you can have the network generate a token that can be used as input to future token generation along with each output token it generates These are called reasoning tokens Initial results with gpt2 are promising You can generalize this to let the network decide when to generate reasoning tokens (I'm unclear on how). There were also multiple lines in the loss graph with reasoning tokens that I d…

Reasoning 1 vs. 3 is the number of reasoning tokens between each "text" token. The 1 reasoning token is exactly what you see in the picture explanation in the article. The generalization comes from making the network predict a and end the sequence only when it predicts a . The training dataset for the upcoming experiment contains examples like: """ Q: What is 3+2? A: 3 + 2 is equal to ... 5 """

Wasting two tokens on start/end reasoning seems expensive to me (a priori)

I am curious what that would yield though - in some ways that would be the most fun to analyze (when does it think a lot??)

I would also be curious to see at what point you see diminishing returns from reasoning tokens (eg a 1:10 ratio? More?)

Re: Self-reasoning tokens: teaching models to think ahead

#25

Post by refulgentis »

sdwr wrote: post #17
wantsanagent wrote: post #10

"The second token, however, duplicates the input of the first one and does not receive a gradient "answer" from the very next token, only from future tokens; ..." This formulation doesn't make a lot of sense to me. I get the motivation here but what you're trying to implement is a working memory. Because transformers have perfect retrospective memory within their context window any generation which can be done direct…

As I understand it, this isn't trying to implement actual memory in the form of a cache, but instead some kind of wishy-washy memory-lite. I'm talking out of my ass here, but I feel like real memory shouldn't be that hard to implement on top of chatGPT. Just run it twice per query, the first time as an internal query that fetches from a memory store. The budgeting part would be interesting. How many tokens of the mai…

I'll call it "not even wrong" :P here, they're putting it in the model, you're describing a common bit of working with LLMs across memory / RAG / etc.

Re: Self-reasoning tokens: teaching models to think ahead

#26

Post by rullelito »

benreesman wrote: post #23

With a little engineering rigor we could do a push-down automata with semantics Girards-Reynolds constrained around polymorphism.

Utilizing Girard-Reynolds constraints on a polymorphic push-down automata fundamentally misconstrues both computational topology and dynamic system semantics..

Re: Self-reasoning tokens: teaching models to think ahead

#27

Post by HarHarVeryFunny »

earslap wrote: post #22
HarHarVeryFunny wrote: post #14

Earlier quoted context omitted.

People are considering that sort of beam-search approach - this is what they call "tree of thoughts" - generate a branching tree of alternate continuations, then pick the best one based on some criteria. This doesn't seem an ideal approach though, since it amounts to generating a bunch of shallow responses and picking the best, rather than the preferred thinking more deeply before generating. It's not the same as a c…

Ah yes, I totally agree. I was inspecting the method as a stopgap solution (especially because it does not require retraining or any other special tricks) until researchers figure out "planning" in a broader sense. It is very inefficient otherwise, but in the meantime, is just simple sampling with a couple parameters to tune from the output softmax the best we can do? is there no low hanging fruit there?

I suppose the closest alternative to planning ahead (considering alternatives before taking any action - in this case generating tokens) is getting it right the first time, which is only really possible in cases of highly constrained circumstances (prompts) where the model saw enough similar examples to predict the same correct/preferred response. So, to that extent, I suppose better prediction - bigger model, more/better training, etc, reduces the need for planning a bit. Architectural changes, such as adding working memory, that boost predictive power, would also help.

But, yeah, hard to see too many alternatives.

1) Get it right first time (not always possible)

2) Don't plan, but at least consider a bunch of poor alternatives - tree of thoughts

3) Actually implement planning

Re: Self-reasoning tokens: teaching models to think ahead

#28

Post by sdenton4 »

lucidrains wrote: post #21
lucidrains wrote: post #20

Earlier quoted context omitted.

yes, it is a stop gradient mask on the attention matrix, iiuc. worth trying

could even try it with a fraction of the attention heads, instead of introducing new tokens

An important piece here is that there's still a training signal making it to the makes weights. See SimSiam for a similar example.

Re: Self-reasoning tokens: teaching models to think ahead

#29

Post by lucidrains »

sdenton4 wrote: post #28
lucidrains wrote: post #21

Earlier quoted context omitted.

could even try it with a fraction of the attention heads, instead of introducing new tokens

An important piece here is that there's still a training signal making it to the makes weights. See SimSiam for a similar example.

indeed, simsiam is a great example of the effectiveness of using stop gradient
Post reply on HN