Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

github.com

Post reply on HN
Print view View original

151–160 of 165 posts

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#151

Post by m3h »

Indeed, it looks like my work has suffered from the clustering issue as well:

  reasoning_output_tokens    count    percent
  ━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━  ━━━━━━━━━
                         0      873    28.5948
  ─────────────────────────  ───────  ─────────
                         8       64     2.0963
  ─────────────────────────  ───────  ─────────
                         9       60     1.9653
  ─────────────────────────  ───────  ─────────
                        11       54     1.7688
  ─────────────────────────  ───────  ─────────
                       516       48     1.5722
  ─────────────────────────  ───────  ─────────
                        12       45     1.4740
  ─────────────────────────  ───────  ─────────
                        10       43     1.4085
  ─────────────────────────  ───────  ─────────
                        17       40     1.3102
  ─────────────────────────  ───────  ─────────
                        13       38     1.2447
  ─────────────────────────  ───────  ─────────
                        14       36     1.1792
Created a script for this: https://github.com/thehappybug/codex-reasoning-token-check

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#152

Post by redml »

if these ai companies want to be taken seriously as being productivity tools then they're going to have to stop with these ab tests and forcing unproven features onto everyone. it's bad enough that ais are inherently unpredictable in quality of output, but these kinds of changes just make things worse.

anthropic at least does have a latest and stable channel, as the other day they pushed something irritating that would skip question asking phase if you didn't reply in 60 seconds, and it broke my multi terminal workflow. like I don't know what their product people are thinking when they push this kind of stuff, but it made me switch to stable

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#153

Post by m3h »

m3h wrote: post #151

Indeed, it looks like my work has suffered from the clustering issue as well: reasoning_output_tokens count percent ━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━ ━━━━━━━━━ 0 873 28.5948 ───────────────────────── ─────── ───────── 8 64 2.0963 ───────────────────────── ─────── ───────── 9 60 1.9653 ───────────────────────── ─────── ───────── 11 54 1.7688 ───────────────────────── ─────── ───────── 516 48 1.5722 ───────────────────…

When I reviewed the conversations affected by this issue, they did not always align with my feeling of "degraded output".

Some were definitely below par, and I recall having to iterate on the generated code more than I wanted to. However, it is only true for a very small number of conversations.

So we're looking at a small set of affected conversations, and even within that small set, only a few will have degraded output, likely because the model can compensate for the reasoning defect over the long conversation.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#154

Post by 7thpower »

smoe wrote: post #51
tunesmith wrote: post #34

Earlier quoted context omitted.

How do you mean OpenAI lets you use your own harness? I'm under the impression that a custom harness requires the OpenAI SDK, which requires api tokens rather than plus/pro accounts.

Not just harnesses, you can even use the subscription in CI/CD. That, plus the fact that web chat does not count toward the same limits, is why I think the Codex personal plan is easily 10x the value of Claude Code. https://developers.openai.com/codex/auth/ci-cd-auth

How are you using it in ci/cd?

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#155

Post by nsingh2 »

m3h wrote: post #153
m3h wrote: post #151

Indeed, it looks like my work has suffered from the clustering issue as well: reasoning_output_tokens count percent ━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━ ━━━━━━━━━ 0 873 28.5948 ───────────────────────── ─────── ───────── 8 64 2.0963 ───────────────────────── ─────── ───────── 9 60 1.9653 ───────────────────────── ─────── ───────── 11 54 1.7688 ───────────────────────── ─────── ───────── 516 48 1.5722 ───────────────────…

When I reviewed the conversations affected by this issue, they did not always align with my feeling of "degraded output". Some were definitely below par, and I recall having to iterate on the generated code more than I wanted to. However, it is only true for a very small number of conversations. So we're looking at a small set of affected conversations, and even within that small set, only a few will have degraded ou…

I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle.

There seems to be something wrong with the "commentary" channel related intermediate updates, maybe the model gets confused about what's an intermediate update vs what's the final answer? [1]

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#157

Post by lmwnshn »

walthamstow wrote: post #134
lmwnshn wrote: post #107

Earlier quoted context omitted.

And although Codex Desktop is not open source (AFAIK), it does expose CDP and appserver which allowed me to build a Greasemonkey-like plugin system on top of it. It's surprisingly powerful, you can add transcript annotations, programmatically control the side panels, integrate a native-looking account switcher, and so on.

This sounds really cool. Do you have the source published anywhere?

Yup! MIT licensed, do whatever you want, you can also just use it directly with nodejs installed through `npx clankerbend` [0]. The screenshot on the github repo links you to a YouTube demo.

Warning: I use this daily myself for work, but beyond me arguing with it to get the CDP architecture, this was completely vibe-coded. I mostly use vim mode (jump to line, jump to prev/next user message) and the account switcher.

[0] https://github.com/onewillai/clankerbend

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#158

Post by zuzululu »

nsingh2 wrote: post #149
cageface wrote: post #90

Earlier quoted context omitted.

Verified this locally myself. Thanks for the concrete test. I guess it's time to give Claude another try.

This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1]. I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result. [1] https://github.com/o…

yea i just tried it and it works!!! i dont know why it works but now gpt 5.5 feels exactly like how I remembered it a month and half ago.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#160

Post by solarkraft »

wongarsu wrote: post #129
mohsen1 wrote: post #102

Earlier quoted context omitted.

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

At least there should be a tool call that's the equivalent of saying "wow, this is more complicated than I thought". Humans are also often prone to under-allocating reasoning time and coming to wrong conclusions because their reasoning ends up too shallow. But the best humans are great at mentally mapping the problem space and readjusting on the fly

Deepseek regularly does this for me, at least when actively exploring something (multi-turn). It massively varies its reasoning budget based on the task.
Post reply on HN