Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

github.com

Post reply on HN
Print view View original

121–130 of 165 posts

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#121

Post by josephernest »

You can use this small Python script to display an histogram of `reasoning_output_tokens` in your past Codex sessions. I do see a spike at 516 indeed.

  import os, glob, re
  import matplotlib.pyplot as plt
  vals = []
  for f in glob.glob(os.path.expanduser(r"~\.codex") + r"\**\*", recursive=True):
      if os.path.isfile(f):
          try:
              s = open(f, "r", encoding="utf-8", errors="ignore").read()
              vals += [int(x) for x in re.findall(r'"reasoning_output_tokens"\s*:\s*(\d+)', s)]
          except Exception:
              pass
  plt.hist(vals, bins=200, range=(0, 5000), weights=[100 / len(vals)] * len(vals))
  plt.xlabel("reasoning_output_tokens")
  plt.ylabel("%")
  plt.show()

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#122

Post by rightbyte »

miki123211 wrote: post #92
postalcoder wrote: post #59

Earlier quoted context omitted.

You still have to worry about misconfigured local models. Even the professionals get it wrong, which is why local model performance is uneven across providers.

And to add insult to injury, some providers will ride on the good reputation of some local model, selling you a terrible quant instead. With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5. You can't say that about glm for example.

Sam Altman, good reputation?

> With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5.

How do we know that? The "orchestration" layer probably forwards to different levels of quantization. And it seems tempting to make some sort of load balancer with adaptive computation effort.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#123

Post by tyingq »

> reasoning-token clustering at 516/1034/1552

Interesting. So 516 probably means initial 512 byte buffer and a 4 byte header. Then 516 + 518 = 1034...so another 512 + 4 byte header + 2 bytes for a linked list ref or similar, 1034 + 518 = 1552, etc.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#124

Post by notfried »

ghosty141 wrote: post #24

Maybe its just bad memory but I feel like 5.3 was the best version in terms of token usage and code quality. 5.5 works better but it just eviscerates tokens.

5.3 was incredibly better than 5.4/5.5. I stuck with it for months after 5.4 was released, and kept testing 5.4/5.5 every now and then but they both were too inconsistent, too rash. I switched to 5.5 a few weeks ago and now regret it, but I am no longer seeing 5.3 as an option to use, only 5.3-Spark, which is trash compared to 5.5.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#125

Post by criley2 »

jatora wrote: post #48
criley2 wrote: post #37

Earlier quoted context omitted.

See, this is part of the confusion. There is no such thing as "GPT-5.5-codex". The last codex-branded model was "GPT-5.3-codex". Starting with "GPT-5.4" the main model handles agentic engineering and they did not release a coding model. Both the web harness and codex app/cli use "GPT-5.5".

haha woops. guess im the chinaman now

在英语中，我們会说“Chinese”。

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#126

Post by aspenmartin »

mohsen1 wrote: post #102
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

Right now we have a LOT of band aids. You want to optimize compute and thinking to a particular problem, sort of like we do. Yes you cannot perfectly predict this but you can do decently well and save a ton of tokens at the cost of this band aid being sort of leaky and gross.

But the larger problem is sound, and the answer is something jointly optimized (idk how they do the routing) but it’s hard to shoehorn it into the current paradigm.

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#127

Post by benjiro29 »

AmazingTurtle wrote: post #108

It's funny, they sell you a subscription for frontier models, then over time begin to nerf them rapidly and no one talks about it. Should give me a discount when they reduce reasoning effort silently on the server side! But on the other hand, I've been using 5.5-high on a daily basis in multithreading workflows, i.e. in parallel. I'm barely exhausting my weekly limits. I can't even Human-as-a-Service fast enough to c…

> they sell you a subscription for frontier models, then over time begin to nerf them rapidly and no one talks about it.

People talk about it all the time. Just check some of the dozens of forums where its non-stop complaining about nerfs, limit nerfs, performance issues etc...

Is hard to prove that any downgrade is a effect of being deliberately served a lower class model / lower quant, or whatever. Or the "optimizations" hurting the models performance.

The TOS allows for those service "optimizations", so legally, nobody has a foot to stand upon. Like when OpenAI or was it Anthropic played with the cache, this to free up more server resources, only to later discover that its gutted the long term context behavior, and heavily degraded the models as context grew.

If you want 100% guaranteed the same performance/behavior, you need to run a model yourself (be it rented GPUs online or your own local setup). But its going to cost you a lot more ...

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#128

Post by bentoner »

nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I've made a passive workaround: a pair of Codex CLI hooks that detect the truncation from the local session transcript and warn — in the TUI at turn end, and via a message injected into the model's context on your next message.

See https://github.com/bentoner/codex-516-hook

Re: GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance

#129

Post by wongarsu »

mohsen1 wrote: post #102
nsingh2 wrote: post #16

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result. Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to wor…

I have a philosophical problem with adaptive thinking. It’s a dumb guess for how much thinking budget to allocate ahead of thinking. At least in the context of LLMs there is probably no way of knowing how much thinking (token generation) is needed. The problem space is infinity vast, similarly of two prompts is not going to help any LLM decide how much thinning is needed. Models already stop thinking before hitting t…

At least there should be a tool call that's the equivalent of saying "wow, this is more complicated than I thought". Humans are also often prone to under-allocating reasoning time and coming to wrong conclusions because their reasoning ends up too shallow. But the best humans are great at mentally mapping the problem space and readjusting on the fly
Post reply on HN