[flagged]
We must create new stories for good so that evil has something to steal! We're running out of unironic irony!
How Compaction Works in Pi
21–30 of 100 posts
Re: How Compaction Works in Pi
#22Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages.
For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.
Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)
Re: How Compaction Works in Pi
#23OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
Re: How Compaction Works in Pi
#24I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
Sounds like you might like subagents. Agent > subagent receives agent context (presumably cached)->tool call->compact/summarise->return to main agent
[dead]
Re: How Compaction Works in Pi
#25I expect Pi is mostly used with OpenAI plans, and OpenAI has a dedicated compaction endpoint you should probably be using with their models instead of a compaction prompt.
There is a compaction routing plugin, as well as a general OAI compatibility plugin that simulates some of their harness features (ie. code_mode/batching tools) that also includes their compaction.
That said, from reading the pi subreddit I don't see people referencing these much.
Re: How Compaction Works in Pi
#26TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information. See: https://github.com/earendil-works/pi/blob/main/packages/codi...
Can also be the same model. The main thing is that it uses a separate, clean context window.
Re: How Compaction Works in Pi
#27I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a turn, seriously driving up cost.
Re: How Compaction Works in Pi
#28Opencodes dynamic context pruning works by labeling tools and chat and the rest and the agent can collapse and expand summaries. I get it into 1M+ routinely on local models with operations between 50k-85k
That dynamic pruning sounds bad for prompt caching though.
Re: How Compaction Works in Pi
#29Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
I think there are a lot of strategies that will open up when costs come down but right now you take a hit on cache rate and thus costs every time you do anything other than wait until the last minute to compact
Re: How Compaction Works in Pi
#30Was expecting the article to go more in-depth. Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
There is no chain, just a lossy summary that includes a summary of any previous summary along with the fresh messages.