Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI reduces Codex Model Context Size from 372k to 272k

github.com

Post reply on HN
Print view View original

61–70 of 194 posts

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#61

Post by jmalicki »

tekacs wrote: post #47
eagerpace wrote: post #39

Earlier quoted context omitted.

The best way around this for me is planning a project well ahead of time and using subagents to execute individual tasks while the primary agent acts as a PM.

That isn't really a way around it for the kinds of things I'm talking about. The whole point is that getting all of the relevant context to even orchestrate it takes up most of the model's 272K.

For those kind of tasks, I do hierarchical planning, and then parts of the plan can focus one agent on keeping a smaller amount of pre-determined context when orchestrating.

Think: A VP of engineering has some super high level goal: "build a new product to beat our competitor in this new market". It then filters down and various levels of reports build more and more detailed yet more narrow pieces of planning/orchestration. They don't make it work by keeping all of the context in their head and micromanaging.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#62

Post by thejazzman »

jmalicki wrote: post #58
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.

This used to be really effective and cheap, now it lights my quota on fire, even after I fix the config and restore smaller subagents instead of like-agents

Multiple (slow) conversations seems to be the efficient path for me the past week or so

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#63

Post by ljm »

jmalicki wrote: post #58
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.

How does that work out in terms of usage? I imagine that, unless you're on the highest tier subscription or have a no expense spared approach to agentic coding, then you're going to hit a wall pretty fast.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#65

Post by jeremyjh »

tekacs wrote: post #33
planckscnst wrote: post #31

Earlier quoted context omitted.

I agree. Compaction sucks, so I made tools that let the LLM selectively delete (and recall if needed) chunks of its context. You might want to try context bonsai if you're routinely hitting the auto-compaction wall. https://github.com/Vibecodelicious/context-bonsai-agents

I've implemented a similar approach – although I'm surprised not to see mention of cache prefix busting in there!

Yes the idea is cool but this could really hammer usage, especially just leaving it up to the agent to decide when to do it. I'm not surprised though, considering the github account is named "Vibecodelicious" and became active in December.

Poking around in the repo the whole implementation is an unsupervised LLM fever-dream.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#66

Post by yearolinuxdsktp »

hakanderyal wrote: post #54

I have rule files that guides the agent towards my coding standards, code style, house rules etc. They alone cost 60-80k tokens, and they are the backbone of my system that prevents slop. Pre 1M context, I had to build complicated tooling to re-include the relevant docs to the context upon compaction, which relied on unstable transcription file format, which was a pain to maintain. With 1M context I deleted all of th…

Try running your rule files through an LLM for optimization. 60k-80k tokens is massive.

Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs.

I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better.

I had one critical rule I was maintaining about searching the codebase using a structural index/graph and not grep. Every time the agent missed it, I asked it how to improve the rules. Eventually, I asked the AI to review that rule file and it rewrote it to be 30% smaller, but, crucially, structured to be more understandable by the LLM.

Another helpful thing was to ask AI to review my rules for things it can load on-demand when it works in that area.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#67

Post by quotemstr »

Codex is open source. You can build it yourself. You don't have to use upstream's parameter choices, compaction strategies, subagent heuristics, or whatever. Strange how people act like tweaks to these parameters are mandates when they're really just suggestions relayed to you via git.

For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you go to the /model menu. I didn't like this change, so I undid it locally. Easy peasy.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#68

Post by azuanrb »

tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

Curious, what kinds of tasks do you do that require such a large context window? Anything specific?

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#69

Post by yearolinuxdsktp »

tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

If you’re using the pi.dev harness, I found this tool to be quite good as a compaction alternative: https://pi.dev/packages/pi-blackhole — it keeps a memory so every prior conversation piece is recallable, not lost.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#70

Post by jmalicki »

ljm wrote: post #63
jmalicki wrote: post #58

Earlier quoted context omitted.

Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.

How does that work out in terms of usage? I imagine that, unless you're on the highest tier subscription or have a no expense spared approach to agentic coding, then you're going to hit a wall pretty fast.

It's actually less token usage overall, since the gigantic context that builds up in the main agent is a huge driver of costs.
Post reply on HN