Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-6 Astra, looped transformers, and hidden reasoning

magazine.sebastianraschka.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#141

Post by imtringued »

shawntan wrote: post #52

For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/ Notably Will Merrill's work: https://arxiv.org/abs/2310.07923 As for how universal transformers (looping transformers, but everyone has since forgotten prior work) will affect this, Will Merrill (again) has a…

>The original universal transformers is called "universal" because if you allow for per-token looping decisions, it can theoretically be Turing complete without needing CoT (some nuance here about levels of precision used).

Looping the transformer is just as turing complete as CoT. It doesn't fundamentally grant it any new theoretical capabilities. You could just scale the model into infinity with infinite context window.

Turing completeness doesn't care about the efficiency of the underlying implementation, which is fine in theoretical computer science, but if you have a model with a finite computational budget, you do actually care about the differences between write only tape vs read-write tape and single tape vs two tape. Having a fixed number of registers like a CPU also helps with reducing the number of redundant operations.

We see none of that with looped transformers, maybe we do see a fixed number of registers.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#142

Post by imtringued »

throw3954 wrote: post #8
libraryofbabel wrote: post #2

Everyone interested in LLM internals should read Sebastian. He's great. The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you r…

It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens. To analogize, current transformers run a fixed-length program per step. Any program can be factored into a top-level loop with a fixed-length branching body (an int…

>It’s a little more complicated than that. While looped transformers can be unrolled a fixed number of times to save on memory, if loop depth is determined dynamically between tokens, a single transformer can compute any computable function between tokens.

This is worded so confusingly it might as well tell us nothing, because it is technically true even without looping due to the fact that you still have infinitely growing context and can simulate a standard turing machine using it.

If you loop, you have a fixed capacity memory that you can rewrite but not carry over to the next token, this is different from a non looped transformer where the transformer can only append a new token.

Meanwhile if you have a DEQ with growing context, it is bona-fide turing complete in the most literal sense.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#143

Post by alex_duf »

So if I read this correctly, Astra is not hiding reasoning, and the only technique we know off that hides reasoning is recursive latent reasoning.

Do we know of any major lab or large open source LLM that uses recursive latent resonning? Can't an additional network be trained on that latent thinking trace to decipher what's going on?

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#144

Post by mike_hearn »

andai wrote: post #83

> I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its…

They can spot evaluation awareness because it appears in reasoning tokens.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#145

Post by mike_hearn »

olalonde wrote: post #138
lann wrote: post #132

Earlier quoted context omitted.

Yes.

Container or full blown VM?

I use containers in one context (custom container manager) and a regular UNIX account on bare metal in another.

This isn't intended to stop a model like Astra hacking its way out of course, it's more like guardrails on a staircase.

My personal container manager tool has an intercepting SSL proxy and small Javascripts on the host can rewrite or block HTTP requests. The agent gets its own isolated home directory and can't tamper with mine. Local caches like Maven are mapped read/only with a write layer on top.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#146

Post by jiggawatts »

atomflunder3000 wrote: post #75

I only used Astra while coding a bit so I can't comment on anything else but I have been really disappointed by it. It seems to overengineer really bad and it is also very slow due to it "thinking" too much I feel like. One example is that I asked it to implement a new functionality inside an existing App of mine and if I had written it myself it would have been like a ~50 line diff. Astra took like 10 minutes to wri…

Try Astra on low or at most medium thinking level. Its "low" is better than Sol "high" or even "xhigh", and then it also doesn't overthink as much.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#147

Post by imtringued »

SubiculumCode wrote: post #101

The major concern with looped transformers is that makes it more difficult to monitor model alignment. When more processing occurs within latent space without outputting text, that means less effective, frequent chain-of-thought monitoring, and the potential for greater un-monitored latent-space shenanigan.

This is silly, the entire reason why chain of thought even exists is to let the LLM "think independently" instead of minimizing the deviation from the supervised training sample. It's an intentional scratch pad for intermediate data. The loose monitoring is kind of the entire point.

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#148

Post by program_whiz »

fc417fc802 wrote: post #88
program_whiz wrote: post #62

Earlier quoted context omitted.

Actually, removing CoT might make models safer, because we can analyze the entire landscape of their potential outputs, rather than a point-sample (we'll never know how close we were to "kill all humans"). By inspecting intermediate vector spaces, we can actually get certainty bounds on how safely the model is behaving (or even trending). Wrote about it here: https://substack.com/home/post/p-214402969

I don't see why you have to remove CoT to do that?

Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).

Re: GPT-6 Astra, looped transformers, and hidden reasoning

#149

Post by smusamashah »

andai wrote: post #70

The MSPAINT computer use demo made my jaw drop. I guess it's not too different from the SVG pelicans, in terms of what it's doing, but it's still amazing to see it working in real-time like that.

Author didn't say it was real-time.
Post reply on HN