161–169 of 169 posts
The MSPAINT computer use demo made my jaw drop. I guess it's not too different from the SVG pelicans, in terms of what it's doing, but it's still amazing to see it working in real-time like that.
This one's even better. Using Canva https://x.com/iam_zachi/status/2095992132620136677
Earlier quoted context omitted.
You need to separate memory capacity and bandwidth. Looping decreases memory capacity/FLOP but not bytes loaded/FLOP, since weights need to be loaded again for the 2nd pass. Plus (depending on the method used) capacity required for KV will be that of the equivalent unlooped model (44 blocks) and KV is typically larger than weights at long context.
Why would weights need to be « loaded again » for the 2nd pass? Weights never change at inference time no?
Astra was insane until Monday but something happened on tuesday, now it feels like Sol. I grieve for the lost productivity but i hope they may give us the original Astra back.
I think it's important to have a solid benchmark where you KNOW there's a difference in model performance.
I have one around 3D modeling that models really land in the same space each time I run it. It's visual, and it's super clear. Sol has perpetually generated low quality work regardless of reasoning level. Astra was the first OpenAI model to suddenly leapfrog the pack and generate content that was production ready, beating out any other provider.
I haven't seen Astra regress (yet).
I think, if you want to be consistent and scientific about it, then you'd have to use the models via API and lock to a specific version. Via the subscriptions, you are floating on whatever the latest version is, vendor to vendor.
If you loop an entire transformer model on itself, that seems like by-definition hidden reasoning. If the output of the model is its reasoning trace, and you simply feed that back into the model again at inference time instead of outputting it - then it is by definition hidden (but I would expect you could pull both this trace and a further-down final output trace out)
I really don’t understand how this is misunderstood by people that should know better.
Another way to point out the silliness. Raschka's own argument: his Luna vs Sol point shows that ordinary added depth already shifts computation into latents, and nobody called that hiding.
So if I read this correctly, Astra is not hiding reasoning, and the only technique we know off that hides reasoning is recursive latent reasoning. Do we know of any major lab or large open source LLM that uses recursive latent resonning? Can't an additional network be trained on that latent thinking trace to decipher what's going on?
I realy don’t understand how this is possible.
Earlier quoted context omitted.
I don't see why you have to remove CoT to do that?
Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw p…
How would you do the same if all reasoning is happening in looped transformers? You would have to develop very sophisticated interventions that construct hidden states which you inject into the model instead while it is thinking. Much harder, and much easier for the model to use weird correlations across the hidden state to hide misaligned thought patterns.
(surprising they called it "loop")
Earlier quoted context omitted.
Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw p…
No, this argument doesn't make any sense. With CoT, the model must compress hidden state to actual text and use its scratchpad as a bottlenecked representation of its past thinking. Thus, it is observable and we can tell by the pattern of a CoT what it was thinking to some extent if we do proper interpretability. Change the CoT text, and the model has literally changed the way it was thinking for the next tokens. How…