Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

github.com

Post reply on HN
Print view View original

51–60 of 165 posts

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#51

Post by smoe »

tunesmith wrote: post #34
selectodude wrote: post #25

Earlier quoted context omitted.

At least OpenAI lets me use my own harness. Having to rely on insane PMs letting Claude Mythos go wild on the codebase has not been going well lately.

How do you mean OpenAI lets you use your own harness? I'm under the impression that a custom harness requires the OpenAI SDK, which requires api tokens rather than plus/pro accounts.

Not just harnesses, you can even use the subscription in CI/CD. That, plus the fact that web chat does not count toward the same limits, is why I think the Codex personal plan is easily 10x the value of Claude Code.

https://developers.openai.com/codex/auth/ci-cd-auth

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#53

Post by dannyw »

nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I wonder if testing during different time/days show patterns? For example, whether the short circuiting happens more often during workday peak hours.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#54

Post by gwerbin »

jerezzprime wrote: post #36
siva7 wrote: post #10

Earlier quoted context omitted.

I've switched 3 months ago to Codex because Claude got incredibly stupid. 6 months ago vice versa. It doesn't matter if you use Codex or Claude. Both will fuck with you at some point. Though Codex probably less.

If you use GitHub Copilot you can switch between them in the same session if you want.

Yeah but now that you pay for tokens that's going to be bad for token caching.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#56

Post by dannyw »

ACCount37 wrote: post #30
perching_aix wrote: post #20

Earlier quoted context omitted.

It seems to be an inference engine or agent harness defect/misconfig rather. Not only do the issue details not evidence a willful stealth nerf, they actively suggest otherwise: the root cause is crude, and evidently not particularly stealthy (as it's being reported on by a regular user with independently verifiable, exact details). I don't find "usual user psychosis" particularly fair or tasteful anyhow. You're not l…

"They made the model dumber" on literally the same checkpoint with the same prompt on the same quantization running on the same hardware is a staple of AI complaints. Users are completely incapable of objectively evaluating model quality over time. Which makes it all the harder to notice actual "stealth nerfs", misconfigurations or other technical issues. Because "they made the model DUMBER, for REAL this time" is ba…

How are you so sure that frontier API models are always running the same quant/weights/etc? You think OpenAI and Anthropic are running essentially just vLLM endpoints? Of course not.

Firstly, we know Anthropic has been doing prompt injection into their 1P APIs (not bedrock/vertex AFAIK) for at least a year now. https://old.reddit.com/r/ClaudeAI/comments/1f6hcwo/injection...

This can be verified pretty quickly like OP — count the token metrics, if your context contains classifier-firing terms, you’ll see input_tokens being higher than your input.

So if they’re already doing that, what makes you think it’s just a dumb API, instead of a complicated pipeline filled with trade secrets and optimisations?

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#59

Post by postalcoder »

nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

You still have to worry about misconfigured local models. Even the professionals get it wrong, which is why local model performance is uneven across providers.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#60

Post by [deleted] »

postalcoder wrote: post #59
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

You still have to worry about misconfigured local models. Even the professionals get it wrong, which is why local model performance is uneven across providers.

[deleted]
Post reply on HN