Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI reduces Codex Model Context Size from 372k to 272k

github.com

Post reply on HN
Print view View original

101–110 of 194 posts

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#101

Post by jmalicki »

skybrian wrote: post #37
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

That seems quite different from my design process. I write a plan.md that goes through multiple revisions. The plan is the memory. Restarting and reading the plan again to do another review is a good way to get a different perspective.

I like to actually have hierarchical plans.

Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.

That document gets saved, and has the big picture context.

Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.

The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.

Then have a DAG of stacked pull requests designed for subagents to work through.

All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#102

Post by causal »

dannyw wrote: post #2

This was tweeted about when it happened, with some explanation from Tibo here: https://x.com/thsottiaux/status/2076543065045795309

Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#103

Post by mikhailt »

embedding-shape wrote: post #8
davidkuennen wrote: post #7

I never felt that context size was any issue at all in codex. I don't know how their compaction works, but it just keeps on going as if it has no context limit at all. At least in my experience.

> I never felt that context size was any issue at all in codex. I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago. Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface…

That has nothing to do with context window or the models at all; it's the compute capacity.

Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.

Claude had the same issues when they released their newer models.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#104

Post by cortesoft »

tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#105

Post by mikhailt »

cyb3ralbert wrote: post #4

Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, and most sessions probably don't need anywhere near 372k tokens anyway. Curious if this affects people who were actually relying on the larger window for big codebases.

We'll have to wait because Tibo (OpenAI) said this was a temporary measure and it'd go back to 372k soon.

The issue was more specific to higher token burn rates, not latency.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#107

Post by bredren »

jubilanti wrote: post #81
damsta wrote: post #20

Not a fan of their context compaction and I feel like 1M-token context should minimum today. Each day I see how GPT 5.5 and 5.6 struggle a bit after each compaction before they get to the full speed, sometimes focusing too much on some older steering message that made it into the compacted context.

The fact there is no way to disable auto-compaction like every other coding harness makes me scream. It fires randomly at 10-20%, so really you now only have guaranteed context of 272k * 80%. So many times I've had it work on a large codebase, it is almost done with a task, it needs maybe a 2000 token response, but it drops below 20%, it churns and churns and churns and then I see the dreaded ``Context compacted`` an…

Pardon the plug, but I have built a tool that ingests the state of conversations to a local DB in realtime. It has both macOS and Linux clients.

When context compaction introduces a gap, I use the /total-recall skill to pull prior turns back into context and off it goes.

The tool is free for personal use and has a source available local cloud option to sync convo histories across multiple machines.

http://contextify.sh/docs

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#108

Post by onetrickwolf »

I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway.

I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.

It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#109

Post by CamperBob2 »

cortesoft wrote: post #104
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.

Exactly. Just about every prompt I issue these days generates or edits an .md file.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#110

Post by sdesol »

volker48 wrote: post #97
jubilanti wrote: post #79

Earlier quoted context omitted.

The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me on any codebase > 5kloc. Even worse it seems to fire randomly at 10-20% of context left. So really you have 80% of 272k as real usable context. Compaction kills my sessions, it hallucinates and is worse than starting fresh. I've had enough times screaming at my computer…

Have you tried using another harness like Pi? You can easily turn off auto compaction and if you leave it on you can use /tree to go back before the compaction if you like.

> you can use /tree to go back before the compaction

This is really a killer feature in my opinion. I'm currently working on pi brains extension that is designed to solve the compaction issue I have, which is, I don't know what the AI knows after compaction.

https://gitsense.com/screenshots/pi-inspect-overview.png

The screenshot shot above contains what I call a compaction capsule which contains all the messages/events at the point of compaction. The "Files" section in the right side panel shows me how many files were read/modified since compaction.

After compacting, I can easily tell what Pi use to know and use `/tree` to go back in time.

I still wouldn't compact more than 5 times before starting a new session since compaction messages persists in Pi.

Post reply on HN