Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI reduces Codex Model Context Size from 372k to 272k

github.com

Post reply on HN
Print view View original

151–160 of 194 posts

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#151

Post by behnamoh »

jubilanti wrote: post #79
tekacs wrote: post #26

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately. Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine? The lack of long context is the main reason that I still end up using Anthropic. The worst is whe…

The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me on any codebase > 5kloc. Even worse it seems to fire randomly at 10-20% of context left. So really you have 80% of 272k as real usable context. Compaction kills my sessions, it hallucinates and is worse than starting fresh. I've had enough times screaming at my computer…

> The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me

Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#153

Post by Havoc »

onetrickwolf wrote: post #108

I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway. I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modula…

Yeah I also try to compact/restart at 250k. I noticed though that it tracks with how big the project is so I guess the people needing more are just working on bigger projects

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#154

Post by ilc »

anukin wrote: post #132
jwitthuhn wrote: post #120

Earlier quoted context omitted.

Claude does the same when you hit the context limit, but the difference is that claude's context limit is 1m and Codex's is 272k now. Claude lets you get a lot more done before you have to worry about the start getting compacted away.

Have you observed any decrease in performance at context higher than 200k? That used to be one of the major issues why clearing or compacting context when the context nears 150 to 170k is usually preferred.

Opus: Yeah, it can be a little fuzzy at 600-700k, I prefer it 400K and under in general. But, I've had usable sessions to 850k.

It really depends on the session, and what you are doing how far you can push it.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#155

Post by searealist »

noname120 wrote: post #73
quotemstr wrote: post #67

Codex is open source. You can build it yourself. You don't have to use upstream's parameter choices, compaction strategies, subagent heuristics, or whatever. Strange how people act like tweaks to these parameters are mandates when they're really just suggestions relayed to you via git. For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you g…

Not true. Try to increase the context window and you will be greeted with an error when you reach the real limit enforced server-side.

The models themselves support up to 1M. You are just charged more for all context over 400k. The 272k limit is just a client limitation to ensure you can never go over 400k since the maximum output size of the model is 128k.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#156

Post by wonnage »

fandorin wrote: post #49

Does the context size really matter that much if a “dumb zone” starts around 120k-150k anyway?

Not sure if this is still up to date (2023), but https://arxiv.org/abs/2307.03172 shows that performance degrades mostly in the middle of the context.

Anecdotally I've been stuck in that situation of being at 400-500k tokens and "just one more prompt bro" will get the task done, and I appreciate not having to wait through a compaction. If anything, keeping the bloated context helps with accuracy at the expense of speed in these cases.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#157

Post by wonnage »

yearolinuxdsktp wrote: post #66
hakanderyal wrote: post #54

I have rule files that guides the agent towards my coding standards, code style, house rules etc. They alone cost 60-80k tokens, and they are the backbone of my system that prevents slop. Pre 1M context, I had to build complicated tooling to re-include the relevant docs to the context upon compaction, which relied on unstable transcription file format, which was a pain to maintain. With 1M context I deleted all of th…

Try running your rule files through an LLM for optimization. 60k-80k tokens is massive. Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs. I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better. I had one critical rule I was maintaining about searching the codebase usi…

AI cannot tell you why it did something. If you ask it why it didn't do X even though your prompt said so, it can come up with some plausible sounding explanation, which might even be correct, but there's a fundamental impedance mismatch between giving you the most plausible sounding answer and actually grounding that in knowledge of how it generated that answer.

TBF having a 60k personal style guide is also a bad idea, people get borderline AI-psychosed about how their special prompts are really steering the AI when in reality there is no way to evaluate this stuff. You invariably end up trying to define your style in terms of what it is and it isn't, it's like trying to define "red". "follow the style of my 5 most recent PRs" works pretty well.

Re: OpenAI reduces Codex Model Context Size from 372k to 272k

#160

Post by chaos_emergent »

causal wrote: post #102
dannyw wrote: post #2

This was tweeted about when it happened, with some explanation from Tibo here: https://x.com/thsottiaux/status/2076543065045795309

Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?

The chart makes sense and is describing the cumulative cost of a trajectory. Cache tokens are created when a trajectory’s prefix is used more than once. A larger pre-compaction context window means that a greater number of cache tokens are used per turn, and a larger number of turns are completed before compaction runs. So you get a cumulative cost that grows quadratically until the compaction event.
Post reply on HN