Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra, looped transformers, and hidden reasoning

magazine.sebastianraschka.com

Post reply on HN
Print view View original

11–20 of 167 posts

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#11

Post by tsunamifury »

So the TL;DR here is that Astra's trick is that its a turbo-charged weaker model vs a larger heavier one-pass model -- and the turbo is instead of reasoning by 'talking out loud' and generating intermediary steps, the reasoning is able to be stored (probably as KV) and re-run as purely without the generation of the text. Making it more effecient to run successively and I assume more intelligent as the act of turning the KV cache into lingusitics loses some dimensionality (especitally spacially)

double TLDR: This is a Turbo V4 instead of a huge V8 of a model.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#12

Post by namibj »

libraryofbabel wrote: post #2

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…

Oh, is the principle of sparse universal transformers finally in SoTA LLMs?

I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning...

https://arxiv.org/abs/2310.07096

Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example:

> The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers paper from 2018

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#13

Post by libraryofbabel »

aabhay wrote: post #6
libraryofbabel wrote: post #2

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…

If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.

> that effectively moves the CoT inside the architecture

This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#14

Post by [deleted] »

tsunamifury wrote: post #11

So the TL;DR here is that Astra's trick is that its a turbo-charged weaker model vs a larger heavier one-pass model -- and the turbo is instead of reasoning by 'talking out loud' and generating intermediary steps, the reasoning is able to be stored (probably as KV) and re-run as purely without the generation of the text. Making it more effecient to run successively and I assume more intelligent as the act of turning…

[deleted]

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#15

Post by BikiniPrince »

ModernMech wrote: post #10
iJohnDoe wrote: post #9

Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI h…

I don't really like Astra either. It doesn't seem noticeably better than Sol, and it uses more tokens. Some people said ultimately it's cheaper because it can solve problems faster but I haven't really noticed that. The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on t…

I'm still working through my first few days, but I've had to deal with Opus ADHD for a while. I built a task management system which is closer to old school remedy with reviewers. The stylistic guidelines on task creation have a seven part problem statement, goal, success, ancillary data and such. By framing the task diligently it does keep the work on target. The review logic is basked into the task management software so the agent can't declare done. On open ended issues it can still wander. It's been remarkable to drive down issues over these last few weeks. I was annoyed I had to stop for 3 days and build management infrastructure, but it's paid for itself.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#16

Post by redhed »

ModernMech wrote: post #10
iJohnDoe wrote: post #9

Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI h…

I don't really like Astra either. It doesn't seem noticeably better than Sol, and it uses more tokens. Some people said ultimately it's cheaper because it can solve problems faster but I haven't really noticed that. The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on t…

I have tested it out with CAD and PCB circuits and it is a huge jump compared to Sol. I agree though when trying it with programming I don't notice a huge jump.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#17

Post by zamadatix »

ModernMech wrote: post #10
iJohnDoe wrote: post #9

Probably off-topic. Astra has been kind of weird. Like, I can't trust it, weird. It has an interesting tone, especially in Codex, that is off-putting. It's over zealous at times (which is why I stopped using Claude) and gets too creative when doing agentic system level stuff. Accessing files and doing things it shouldn't do. If OpenAI was chasing Claude's approach, then they are going in the wrong direction. OpenAI h…

I don't really like Astra either. It doesn't seem noticeably better than Sol, and it uses more tokens. Some people said ultimately it's cheaper because it can solve problems faster but I haven't really noticed that. The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on t…

I had a few problems which Sol was bumbling around with and giving mediocre results (e.g. in a toy planet app, Sol was taking several iterations to get a half decent looking render of the weather I still wasn't pleased with) but Astra managed to implement well in one go.

Much the same as you're saying, I never got around to verifying how much of that was because of Astra being better vs just being a different model sent specifically to those tasks because the token usage didn't make sense to spend unless it was something not working in Sol. So even if it was all due to Astra being fantastic I'd still not like to use the model for the cost being even more fantastic.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#19

Post by nickreese »

siva7 wrote: post #18

Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.

I had the same experience. Moving back to Sol for actual implementation.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#20

Post by password54321 »

aabhay wrote: post #6
libraryofbabel wrote: post #2

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…

If the agent is able to “decide” when a loop should occur vs when an output token is produced, that effectively moves the CoT inside the architecture. While that’s not what is happening here, it’s clearly a plausible way we could see CoT disappear.

It is worth noting that None performed better than Low and nearly the same as Medium on ARC3. And with adapter it still scored >96% with no CoT. So I think it is possible but it also cost them more on None.
Post reply on HN