Ampcode used a handoff feature for a while that I found genuinely useful [1] and then they removed it. Anecdotally, I felt it worked better than compaction.
How Compaction Works in Pi
51–60 of 100 posts
Re: How Compaction Works in Pi
#52I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
I mean, not to be flippant but can't you just prompt the agent to write a file as you're getting closer to the compaction limit? I tend to just go to roughly 50-70% context utilization and then tell the agent to summarize the conversation and save it to a file, manually /clear, then say let's continue that last conversation. You can inspect the summary first and make any changes.
Re: How Compaction Works in Pi
#53I don't like that it throws away the whole KV cache when compacting. It costs a cache miss of the whole conversation length, and that's a waste of time and money. LLMs are perfectly capable of summarising the conversation without a new system prompt.
How the summarizing of the conversation happens in an LLM?
Instead of prompting "Summarize: $SESSION" you submit "$SESSION Summarize what has been done so far".
This way you have a prefix cache hit on the session and don't pay any more than for a regular prompt.
Re: How Compaction Works in Pi
#54Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
Potentially: remove thinking blocks, and keep the rest. At least this would ensure that the entire context of the conversation is still there, and anything said isn't lost.
Having a second model also iterate the resulting messages and remove low-value tool calls could also be interesting. Especially failed calls which add no value.
Re: How Compaction Works in Pi
#55In my experience, the best approach to compaction is to never get to the point where you need compaction and to generally stay below about 30% context window utilization. Even for long agentic workflows this can be accomplished for quite a while, much longer than most people might think. Here's what I do for each of my sessions: 1. For asides, off-topic work, or repetitive work that has already been done in the sessi…
What is that 30% number based on? Surely that's a model specific limit, and is based more on the absolute token length, not percentage, right? I'm not sure it makes sense for e.g. Opus 0.2M and Opus 1M to both degrade at 30% of their respective context lengths.
Re: How Compaction Works in Pi
#56Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
Wouldn’t that invalidate kv cache and you wouldn’t benefit from much cheaper cache read?
Re: How Compaction Works in Pi
#57My summarization creation functions over batches of 50 messages, and I don’t often lose important context any more. The loss comes from trying to stick a whole conversation in a single compaction request (at least in my case)
Re: How Compaction Works in Pi
#58Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
Wouldn’t that invalidate kv cache and you wouldn’t benefit from much cheaper cache read?
I'm frequently idle for five minutes while I review, at which point the cache has expired
Re: How Compaction Works in Pi
#59OMP changed the default compaction to images ! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
What's helpful about that?
This is one of the most ridiculously awesome findings this year, that changed my mental model of how foundation models work.
It sounds crazy, but it's way cheaper and faster for multimodal models to read text on images than the equivalent text represented as tokens. Text tokens take a lot more bytes to represent internally than you would think. Give it images with rasterized text, and there is no OCR-like intermediate step that turns images into text tokens. It just goes straight into the multimodal context where it's all just linea algebra underneath. The cost is more lossiness than if you gave it the same tokens as text. Which is why you don't just render text prompts and files to read as images.
But compaction is inherently lossy, a summary 1-10% of the original length is going to lose things, by definition. So you keep the things you really need to get exactly right in text token summary, but put a lot more of what you would have just discarded into a PNG. Keeping the past context as text in an image is way less lossy than keeping a text summary of that context.
They peeked inside the layers and found the same neurons were usually activating in the middle layers when fed either text tokens or images of that same text. It just takes the model more time and kv cache to do it via text tokens.
Deepseek was the first to really jump on this: https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...