Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Compaction Works in Pi

earendil.com

Post reply on HN
Print view View original

61–70 of 100 posts

Re: How Compaction Works in Pi

#61

Post by aevv »

hatthew wrote: post #55
errantmind wrote: post #50

In my experience, the best approach to compaction is to never get to the point where you need compaction and to generally stay below about 30% context window utilization. Even for long agentic workflows this can be accomplished for quite a while, much longer than most people might think. Here's what I do for each of my sessions: 1. For asides, off-topic work, or repetitive work that has already been done in the sessi…

What is that 30% number based on? Surely that's a model specific limit, and is based more on the absolute token length, not percentage, right? I'm not sure it makes sense for e.g. Opus 0.2M and Opus 1M to both degrade at 30% of their respective context lengths.

https://www.producttalk.org/context-rot this article summarises several papers that have explored this, and it does seem to be related to both the absolute number of tokens and the % of the window

Re: How Compaction Works in Pi

#62

Post by zmgsabst »

kierangill wrote: post #22

Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…

I do this in my own harness, where a context only exports certain messages when you step out of it — eg, my request for an essay and its final output, while dropping everything from the first draft through various intermediary edits.

That naturally trims a lot of context while removing mistakes from the context to prevent poisoning (ie, every draft but the final negatively contributes in some way — that’s why we edited them).

Re: How Compaction Works in Pi

#65

Post by charcircuit »

pjm331 wrote: post #29
kierangill wrote: post #22

Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…

I think there are a lot of strategies that will open up when costs come down but right now you take a hit on cache rate and thus costs every time you do anything other than wait until the last minute to compact

You can do things like throwing away or summarizing information from the most recent calls so you fall back to a recent checkpoint that is still cached

Re: How Compaction Works in Pi

#66

Post by jakswa »

UncleOxidant wrote: post #34
jakswa wrote: post #23

OMP changed the default compaction to images ! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.

But what if the model you're using doesn't have image processing capabilities?

I dunno, I didn't read in-depth. Hopefully you don't gotta zoom in with human eyeballs.

Re: How Compaction Works in Pi

#67

Post by tosh »

kierangill wrote: post #22

Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages. For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration. Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations le…

when you look at the compaction prompt: in a sense it is doing that pruning but the llm decides what to prune

Re: How Compaction Works in Pi

#68

Post by pranayVarma0512 »

pornel wrote: post #53
kristianpaul wrote: post #32

Earlier quoted context omitted.

How the summarizing of the conversation happens in an LLM?

Instead of prompting "Summarize: $SESSION" you submit "$SESSION Summarize what has been done so far". This way you have a prefix cache hit on the session and don't pay any more than for a regular prompt.

This seems to be a very intuitive take, but people often miss on the technicalities.

The prompt is assembled as tool schemas, then system instructions, then message history, and caching works on prefixes of that. So the cost of an edit is not its size, it is the size of everything behind it. Edit a tool definition and you have invalidated the system prompt and the whole conversation with it. Edit the tail and you pay for the tail.

I measured this by accident in a multi agent ablation where the only variable was whether the supervisor sent a fixed tool array or a per-task subset to the worker. Runs paying cache creation, out of 120 each: fixed 0 cache creation, per-task subset 58 cache creation. Under a prompt load that was $0.0382 per run against $0.0230, and on a clean context it reversed because there was no prefix worth caching.

Which is the argument for what Pi does here. A pointer is a tail edit. Rewriting is a head edit.

Post reply on HN