Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

github.com

Post reply on HN
Print view View original

131–140 of 165 posts

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#131

Post by irthomasthomas »

mohsen1 wrote: post #102
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

This is what I do in llm-consortium. An arbiter evaluates the response(s) and decides if more iterations are needed. You can also loop until a minimum confidence threshold, but self-reported confidence isn't a great metric.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#132

Post by [deleted] »

josephernest wrote: post #121

You can use this small Python script to display an histogram of `reasoning_output_tokens` in your past Codex sessions. I do see a spike at 516 indeed. import os, glob, re import matplotlib.pyplot as plt vals = [] for f in glob.glob(os.path.expanduser(r"~\.codex") + r"\**\*", recursive=True): if os.path.isfile(f): try: s = open(f, "r", encoding="utf-8", errors="ignore").read() vals += [int(x) for x in re.findall(r'"re…

[deleted]

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#133

Post by [deleted] »

jdiff wrote: post #117
andy99 wrote: post #103

Earlier quoted context omitted.

This isn’t just em-dashes—it's the empty phrase that includes both whatever the contrastive construction is called and an “Honestly”. It might have been human written but the density of LLM flags is undeniable.

Honesty has become a big tell, especially if its brutal. I'm not sure if it's getting worse or if I'm becoming more sensitive, but if I'm being brutally honest I can barely stand using LLMs anymore with their tone and their cliches. It's a horrible nightmare fusion of socmed influencer and LinkedIn hustler that sounds freakish even if it were human, and the density of tropes that are accumulating feels ridiculous. Di…

[deleted]

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#134

Post by walthamstow »

lmwnshn wrote: post #107
miki123211 wrote: post #93

Earlier quoted context omitted.

OpenAI doesn't require that; only Anthropic does. OpenAI's harness is fully open source[1], and (AFAIK) doesn't come with any kind of signed-build request integrity verification like Claude does. And by that logic, if you're allowed to use their API with a fork of Codex that you yourself compiled, there's nothing stopping you from making some other harness act like such a fork. [1] https://github.com/openai/codex

And although Codex Desktop is not open source (AFAIK), it does expose CDP and appserver which allowed me to build a Greasemonkey-like plugin system on top of it. It's surprisingly powerful, you can add transcript annotations, programmatically control the side panels, integrate a native-looking account switcher, and so on.

This sounds really cool. Do you have the source published anywhere?

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#135

Post by fhars »

inigyou wrote: post #120
zuzululu wrote: post #61

this explains so much why gpt 5.5 has been so bad lately it was really puzzling why it struggled so much where when it first came out it was one shotting stuff totally amazing, i tried the prompt that will tell you if your plan is degraded: codex exec --json --skip-git-repo-check --ephemeral -s read-only --disable memories -m gpt-5.5 -c model_reasoning_effort=high "Do not use external tools. A black bag contains cand…

The correct answer is 29, right? You could draw all the watermelons and all the round pieces before drawing a star piece. So the model never gets it right, but it does when listing the cases exhaustively?

No, the answer is 21. You ignored the "Shape is distinguishable by touch before drawing".

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#136

Post by [deleted] »

lhl wrote: post #42
tunesmith wrote: post #34

Earlier quoted context omitted.

How do you mean OpenAI lets you use your own harness? I'm under the impression that a custom harness requires the OpenAI SDK, which requires api tokens rather than plus/pro accounts.

https://x.com/thsottiaux/status/2058071172361998482 "A little secret. About 5% of our production traffic is on the Pi harness, about another 5% is on OpenCode. Reminder you can use your ChatGPT account in a flourishing set of other tools. We’ll continue to make Codex awesome, but you have options."

[deleted]

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#137

Post by rahidz »

cbg0 wrote: post #82
linzhangrun wrote: post #81

Earlier quoted context omitted.

No, the decline in GPT-5.5's performance over the past few weeks is clearly noticeable.

Doesn't look like it: https://marginlab.ai/trackers/codex/

So what are we to make of the two items:

- This tracker not showing any visible degradation. - Clearly incorrect answers being reported due to truncated thinking.

Is the tracker not measuring 'simpler' tasks that might get auto-sent to "low reasoning hell" even on high/xhigh? Is the clustering not actually causing reasoning misses in real-life coding, or not enough of a negative effect compared to the improvements made elsewhere? Something else?

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#138

Post by ComputerGuru »

Already reported (not as thoroughly but still quite detailed) two weeks ago and silently “closed as not planned” (keep in mind that the specific reason might be an artifact of GitHub workflow/UX and not actually the intended reason) without a acknowledgement or a response.

https://github.com/openai/codex/issues/29353

What even is the point of a public-facing bug tracker “for devs, by devs” when this is how reports get treated? Might as well use Apple’s Feedback Reporter that routes to /dev/null instead.

Anyway, I find it near impossible to see how this wasn’t already caught and flagged internally – it’s not a subtle pattern. Certainly they are at the very least collecting and graphing reasoning tokens vs model vs effort” and such an obvious spike at (multiple) single stops (not even distributed over a narrow range) should have been an immediate statistical red flag… which leads me to believe (combined with the fact the previously reported issue was closed without comment) that they’re at least internally aware of this behavior even if it’s not necessarily an intentional side effect of some internal forcing metric.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#139

Post by chazeon »

cbg0 wrote: post #87
chazeon wrote: post #70

Even without stats i know it went bad. In the pass two month barely can do any good scientific writing lately, which of course rely on reasoning. It just writing for gods sake. And it show how far we are from AGI.

This is an intermittent issue, you should still be able to get your work done. 5.5 was released two months ago so perhaps you're using 5.5 wrong and some things that worked in 5.4 require tweaking your prompts?

Well, when 5.5 first came out, it was kind of OK, but now it's almost noticeably worse. It can be done, but requires a lot of effort, just more and more round, and actually, the Gemini Pro on the web (which should be 3.1 Pro) is actually doing a more stable job.

The thing that I ask it to do is like take X and Y paper into Z paragraph --- a not-so-silly model should think of how information in X and Y are related and how they support the whole article to synthesize this sentence in a way that is coherent to the article, but 5.5 now will just copy the stuff without any reasoning about the relation. Of course, this will cost a lot of tokens and will be obvious if not done. One clear indicator is that in a few rounds you can see the length of the article get bloated to 2-3x undesirably long, which is clearly because it is not analyzing/synthesizing the info.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#140

Post by UltraSane »

mohsen1 wrote: post #102
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

Modern LLMs are nothing but band-aids, starting with the absurd bandwidth of HBM3 RAM that makes them possible.
Post reply on HN