31–40 of 100 posts
LLMs are perfectly capable of summarising the conversation without a new system prompt.
I don't like that it throws away the whole KV cache when compacting. It costs a cache miss of the whole conversation length, and that's a waste of time and money. LLMs are perfectly capable of summarising the conversation without a new system prompt.
Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible. One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while…
It's like asking the questions needed to get the answer you need from the LLM, can be either an answer for a plan/todo or a task to code/change something.
OMP changed the default compaction to images ! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
https://github.com/spott/pi-task-compaction
I’m still playing with it, but it essentially has the model define a region, and mark it with begin_task, and end_task. End_task also requires a fairly hefty summery of the entire region.
It then pulls that entire region and replaces it with the summary. If the model wants it can actually look into the output that was pruned from the conversation, but in practice the summaries have been good enough it hasn’t needed to.
I’ve been using it with 5.6 sol, and even really long sessions have like 6% context used at the end.
I don't like that it throws away the whole KV cache when compacting. It costs a cache miss of the whole conversation length, and that's a waste of time and money. LLMs are perfectly capable of summarising the conversation without a new system prompt.
Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…
I feel like this is kind of subagents, but it doesn't have the clear task -> work -> return result -> die model, rather it's more closely aligned to how I myself multitask, maintaining several long-lived threads of execution, and having to intentionally "forget" about the other ones when it's time to lock in and do real work on the most important one.
Is pi a drop in replacement for OpenCode / Claude Code? I cannot be bothered installing 50 million plugins.
Their approach is minimalism. If you need something pi doesn’t have, the recommendation is to ask pi about it and have it build an extension for you, or look for a plugin.
That said, try it. You may find you didn’t need everything in those ecosystems.
Is pi a drop in replacement for OpenCode / Claude Code? I cannot be bothered installing 50 million plugins.
It is not. Their approach is minimalism. If you need something pi doesn’t have, the recommendation is to ask pi about it and have it build an extension for you, or look for a plugin. That said, try it. You may find you didn’t need everything in those ecosystems.
Not a shill.. I just really like pi.