31–40 of 151 posts
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…
Earlier quoted context omitted.
I don't really like Astra either. It doesn't seem noticeably better than Sol, and it uses more tokens. Some people said ultimately it's cheaper because it can solve problems faster but I haven't really noticed that. The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on t…
I have tested it out with CAD and PCB circuits and it is a huge jump compared to Sol. I agree though when trying it with programming I don't notice a huge jump.
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
I wish someone ran some sort of representative benchmark suite every X days to see if this occurs.
So the TL;DR here is that Astra's trick is that its a turbo-charged weaker model vs a larger heavier one-pass model -- and the turbo is instead of reasoning by 'talking out loud' and generating intermediary steps, the reasoning is able to be stored (probably as KV) and re-run as purely without the generation of the text. Making it more effecient to run successively and I assume more intelligent as the act of turning…
Think how a reverse polish notation calculator with reasonably limited data stack depth could run efficiently with a plain vanilla transformer.
But if you input classic grade school parenthesized infix with a few levels of operator precedence, you are no longer able to just evaluate the expression during transformer prefill. Even if you add a stack depth bound worth it reasoning tokens between any two input tokens as they're processed.
UTs can, at least if run with encoder (unmasked) attention, resolve the task through technically-flexible iteration count that can and will follow the evaluation order of the infix operator tokens of the input expression.
While masked attention unfortunately limits it's powers, the fundamental benefit of separating task-specific-intelligence (an individual expert of an MoE) from the notion of which transformer layer has it pre-digested just right for that task/processing to be done to it, allows for massive reduction in model parameter count. Note this comes at a penalty of parameter activations (inference will take more compute).
It's just that at some point you can't afford to just train more parameters, without suffering overfitting issues/failures-to-generalize.
The architecture decoupling learned weights from when they're activated also helps with generalization to out-of-distribution structures. Think resilience against yoda-speak and such.
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
I wish someone ran some sort of representative benchmark suite every X days to see if this occurs.
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…
It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an int…
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
> We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account.
> No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription.
https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491)