The major concern with looped transformers is that makes it more difficult to monitor model alignment. When more processing occurs within latent space without outputting text, that means less effective, frequent chain-of-thought monitoring, and the potential for greater un-monitored latent-space shenanigan.
GPT-6 Astra, looped transformers, and hidden reasoning
101–110 of 167 posts
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#102The major concern with looped transformers is that makes it more difficult to monitor model alignment. When more processing occurs within latent space without outputting text, that means less effective, frequent chain-of-thought monitoring, and the potential for greater un-monitored latent-space shenanigan.
I'm not sure. That paper from anthropic talked about monitoring j space, presumably those same techniques would work here?
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#103> I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its…
It is still helpful I believe, but your point is well taken. The problem is that we have relatively few tools for monitoring alignment, and longer loops of processing that stay in latent space means less ability to monitor.
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#104The major concern with looped transformers is that makes it more difficult to monitor model alignment. When more processing occurs within latent space without outputting text, that means less effective, frequent chain-of-thought monitoring, and the potential for greater un-monitored latent-space shenanigan.
I'm not sure. That paper from anthropic talked about monitoring j space, presumably those same techniques would work here?
I am no expert, but I think it is this: 1) We have few effective tools at monitoring alignment right now, and chain of thought is one of the more effective. 2) Monitoring latent space may be possible, but I do not think it is even close to being a solved problem, nor whether it is possible at scale and outside of controlled problem areas. 3) Finally, more recursion within latent space may complexify the latent representations, not simplify them.
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#105Earlier quoted context omitted.
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
The most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll pe…
To add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#106If you loop an entire transformer model on itself, that seems like by-definition hidden reasoning. If the output of the model is its reasoning trace, and you simply feed that back into the model again at inference time instead of outputting it - then it is by definition hidden (but I would expect you could pull both this trace and a further-down final output trace out)
No. It’s not at all by definition hidden reasoning. Looping transformers uses additional calculations (repeating layers) to generate a token. Reasoning (in this context) is test time generation of multiple tokens that allow a model to have a scratch pad to refine its thoughts, chain of thought reasoning in other words. Doing the former in no way means that you have to hide the latter. Raschka is right in this post, T…
Interesting. I admit I was largely going on what Mr. Harris said on the episode when responding to several posts.
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#107Re: GPT-6 Astra, looped transformers, and hidden reasoning
#108I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers:
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#109> So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights. From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)
I think it's bottlenecked on memory throughput. Someone else more knowledgeable can verify this.
Re: GPT-6 Astra, looped transformers, and hidden reasoning
#110Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI h…
Astra is definitely weird. It is more capable than Sol, no doubt about that. There are things sol could simply not solve that Astra breezes through.
However for typical low to medium difficulty code, it will often either overengineer stuff, create massive functions instead of organized code, and just write very hard to read code. It literally looks like minified code. Clearly they trained it to reduce the number of output tokens and in turn the code is often atrocious. I'll keep trying Astra but I might actually go back to 5.6 sol for many tasks if I keep getting these results.