Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

arxiv.org

Post reply on HN
Print view View original

41–44 of 44 posts

Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

#41

Post by HarHarVeryFunny »

FuckButtons wrote: post #35
HarHarVeryFunny wrote: post #13

It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that RL is being used to shape high level planning-type direction of the output. It seems that the input layers to a Transformer are necessarily going to be doing the most low level work of syntax -> semantic augmentation starting with things like tagging parts of…

this is explicitly why qwen opted to use recurrent layers as the middle layers of their hybrid model.

AFAIK Alibaba's Qwen3-Next architecture doesn't do anything different specifically for the middle layers - it uses a 3:1 mix of recurrent and regular attention blocks throughout the full depth of the transformer.

The motivation for this mostly recurrent hybrid attention is to efficiently support long context lengths.

Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

#43

Post by godelski »

HarHarVeryFunny wrote: post #13

It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that RL is being used to shape high level planning-type direction of the output. It seems that the input layers to a Transformer are necessarily going to be doing the most low level work of syntax -> semantic augmentation starting with things like tagging parts of…

  > the middle layers of the Transformer that are affected most by RL post-training
This is where you should expect most change in models. The beginning layers need to embed while the later layers will reform the result to the final conditions. The middle is what does all the untangling.

Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train

#44

Post by kmeisthax »

aabdi wrote: post #40
kmeisthax wrote: post #24

Earlier quoted context omitted.

I'm wondering if the big problem is just the lack of recurrent connections in the standard Transformer design, and selective layer duplication is just a weird way to fix the same problem. I have to wonder if it would be possible to deliberately architecture a model to discover and exploit layers worth duplicating at training time. The current model architectures we use have a fixed routing of residuals per layer, fro…

Moe style router?

No. Mixture-of-Experts is for dynamically loading different sets of weights[0] per-token. The idea with MoE is that you can have more weights than fit in RAM without having to actually load everything in RAM.

My idea is more about creating recurrent paths in an otherwise all-forward network. The same number of weights would be loaded, we'd just be routing residuals differently.

[0] Usually multi-layer perceptron / linear portion weights - although maybe someone's tried attention head MOE?

Post reply on HN