Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI reduces Codex Model Context Size from 372k to 272k

github.com

Post reply on HN
Print view View original

121–130 of 194 posts

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#121

Post by keeganpoppen »

tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

sometimes compaction is a death knell and other times it's totally fine. it really just depends on whether the thing you were doing at the very end has high correlation with stuff from early on in the session, or if there was an obvious ~break s.t. you were just on some specific subtask. the situation has clearly gotten a lot better than even 6 months ago-- i think openai handles compaction better in my own experience. but it's exactly the kind of thing that yegge was trying to get at with gastown-- i used to run every session s.t. i could re-animate it every time the compaction boundary came... now, when the compaction boundary comes i am so pot committed that i just hold my breath.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#122

Post by sidewndr46 »

tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

I think I've seen the opposite end of this as well: I get the most value from any LLM when I ask it one very targeted question with a followup. I don't fall outside of any sort of context window sizing when I am doing that. That isn't to say that I don't have a few long engineering discussions that are helpful, but most are very to the point. It helps that I have decades of experience at this point and effectively know what to ask to narrow the scope quickly.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#123

Post by HDThoreaun »

cortesoft wrote: post #104
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.

Yea by the time you get to 300k context its IQ is a solid standard deviation lower anyway, why would you really want to beyond that?

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#124

Post by HDThoreaun »

skerit wrote: post #13

No matter how good compaction is, on some big projects it needs to read a lot of files. In my experience the first 200.000 tokens go FAST, but after that it slows down. Most of my Fable sessions don't go over 500.000 tokens, I don't need to compact once. But when I use Codex a single session has to compact over and over again.

In my experience the only reason it needs to read a lot of files is if your agents.md is deficient. It should only need to read the file it's actually working on, maybe a few others. Everything else should be explained in documentation

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#125

Post by enraged_camel »

onetrickwolf wrote: post #108

I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway. I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modula…

>> I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is.

It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.

My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#126

Post by solenoid0937 »

n4te wrote: post #93
jubilanti wrote: post #81

Earlier quoted context omitted.

The fact there is no way to disable auto-compaction like every other coding harness makes me scream. It fires randomly at 10-20%, so really you now only have guaranteed context of 272k * 80%. So many times I've had it work on a large codebase, it is almost done with a task, it needs maybe a 2000 token response, but it drops below 20%, it churns and churns and churns and then I see the dreaded ``Context compacted`` an…

Use pi. Can't trust OAI or Anthropic to make good tools.

Claude Code is much nicer than Pi, especially for long running autonomous work. Pi might be OSS but it's just not nearly as featureful or agentic as Claude Code.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#127

Post by EsotericSoft »

solenoid0937 wrote: post #126
n4te wrote: post #93

Earlier quoted context omitted.

Use pi. Can't trust OAI or Anthropic to make good tools.

Claude Code is much nicer than Pi, especially for long running autonomous work. Pi might be OSS but it's just not nearly as featureful or agentic as Claude Code.

Pi can do all the same that CC can. OSS is nice but the reason to use it is to control what is happening between me and the LLM. See: OP and many others for many reasons, bloated system prompts, the harness changing day to day, Anthropic shipping terrible and untested [features](https://news.ycombinator.com/item?id=48947776) that don't appear in the changelog, Codex ultra subagents nuking usage, etc. You can tell which harnesses are vibe coded.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#128

Post by astroscout »

skybrian wrote: post #37
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

That seems quite different from my design process. I write a plan.md that goes through multiple revisions. The plan is the memory. Restarting and reading the plan again to do another review is a good way to get a different perspective.

[dead]

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#129

Post by cyanydeez »

bel8 wrote: post #5

That's quite small for my workloads. I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact. Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?

using open models and local AI via llamacpp, I'm compacting (via agent directed) between 55k-85k; going up to 120k is rare unless the agent is actually following a non trivial log tracing or some other thing that absolutely requires keeping large context.

I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.

It mostly works to allow iteration into multiple sub components.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#130

Post by ph4rsikal »

causal wrote: post #102
dannyw wrote: post #2

This was tweeted about when it happened, with some explanation from Tibo here: https://x.com/thsottiaux/status/2076543065045795309

Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?

The blue line (200K context) is lower than the red line (300K context). Indicating that it's cheaper to run blue rather than red over longer cycles.
Post reply on HN