How Compaction Works in Pi
earendil.com
How Compaction Works in Pi
1–10 of 100 posts
Re: How Compaction Works in Pi
#2[deleted]
Re: How Compaction Works in Pi
#3Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.
Re: How Compaction Works in Pi
#4TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information.
See: https://github.com/earendil-works/pi/blob/main/packages/codi...
Re: How Compaction Works in Pi
#5Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.
I found hermes to be really lightweight, though I am on a relatively older version and built a custom plugin to lazily load mcps (that's probably in hermes proper by now). Compared to kilo it seems to consume far fewer tokens.
Re: How Compaction Works in Pi
#6Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).
I pretty much just start a new session whenever i fill the context.
Re: How Compaction Works in Pi
#7Opencodes dynamic context pruning works by labeling tools and chat and the rest and the agent can collapse and expand summaries.
I get it into 1M+ routinely on local models with operations between 50k-85k
Re: How Compaction Works in Pi
#8Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.
I keep my max context really small for personal assistant agents; they don't need it. Especially since compaction keeps anything important around anyway. I use 60k with Pi.
Re: How Compaction Works in Pi
#9The advantage of running local stack is that you can do the compaction at the time of inference, i.e. some tool call runs out of context, you can just pause inference, purge/replace old tool calls with their summaries or just logs by operating directly over tokens on a GPU, rebuilding KV cache (one time prefill hit) and resuming the inference, easily being able to e.g. read 1000 markdowns, each 50k long, in a single LLM call. That's not possible with current agentic harnesses using LLM calls.
Re: How Compaction Works in Pi
#10Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.
One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.