Earlier quoted context omitted.
You could set up checkpoints where you store off the current state, and roll back to those checkpoints?
Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
- use checkpoints to save KV cache before trigger CoT
- trigger CoT, save result as a summary
- go back to previous checkpoint
- instead of generating tokens, add result of CoT summary as input tokens
- continue normally
For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc.
This idea is free.
I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.