Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

github.com

Post reply on HN
Print view View original

161–165 of 165 posts

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#161

Post by selectodude »

solenoid0937 wrote: post #147
lhl wrote: post #42

Earlier quoted context omitted.

https://x.com/thsottiaux/status/2058071172361998482 "A little secret. About 5% of our production traffic is on the Pi harness, about another 5% is on OpenCode. Reminder you can use your ChatGPT account in a flourishing set of other tools. We’ll continue to make Codex awesome, but you have options."

They will only do this while they have no users. Once they are compute constrained they won't let you do this anymore

That’s the beauty of using an open harness. I can jump ship if that ever happens.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#162

Post by nickalaso »

So, I have done a bit of research on this with the writeup+skills+scripts on my personal git: https://github.com/NickalasLight/codex-reasoning-bug-512-tok...

I think it is very interesting that: A. Removing the section in question seems to greatly fix performance on the benchmark candy question. B. Removing the section does not appear to change at all mean reasoning token use or the 512 reasoning token hit problem

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#163

Post by gck1 »

smoe wrote: post #51
tunesmith wrote: post #34

Earlier quoted context omitted.

How do you mean OpenAI lets you use your own harness? I'm under the impression that a custom harness requires the OpenAI SDK, which requires api tokens rather than plus/pro accounts.

Not just harnesses, you can even use the subscription in CI/CD. That, plus the fact that web chat does not count toward the same limits, is why I think the Codex personal plan is easily 10x the value of Claude Code. https://developers.openai.com/codex/auth/ci-cd-auth

I'm never going back to claude from codex, including for the reasons you mentioned, but it must be said that web chat inference on ChatGPT is magic incantation, and I'm almost certain they're not serving the same models there as in Codex.

Claude web definitely feels like it's the same models behind as in API, with much less extra behavior/layers that make it behave differently.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#164

Post by DanielLestrange »

The issue affects _all_ models (inclusive the "hidden" guardian codex-auto-review model), and _only_ happens on the PRO plans (both 5x and 20x). It does not, under any circumstances, happen with the PLUS plan.

There is a probably related difference of CTX windows being different (and significantly larger on PLUS) between the plans, as well as an ominous and not-user-controllable flag `use_responses_lite` which is set to true on PLUS. These two discrepancies between the plans may or may not be the cause for the issue.

Amending the base prompt shows some improvements using the canary test prompts, NOT so during real world tasks however.

The API presents mixed results: from zero failures to 2/5 failures across runs.

These results have been taken from deterministic tests and analysis of historic data on both PLUS, PRO and API accounts. More details visible on the related GH issue in comments.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#165

Post by breakingcups »

mohsen1 wrote: post #102
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

Kind of feels like a variation on P versus NP
Post reply on HN