Really good work here, bravo
Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
21–30 of 44 posts
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#22I still can't believe that LLM encoders aren't unsupervised learned. So much left on the table
They are using Qwen, so this is decoder only.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#23Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#24I'm reminded of this dude who was sitting at or near the top of some kaggle leaderboard simply[0] splicing together some duplicated middle layers and applying a bit of fine tuning [0] not simply
I'm wondering if the big problem is just the lack of recurrent connections in the standard Transformer design, and selective layer duplication is just a weird way to fix the same problem. I have to wonder if it would be possible to deliberately architecture a model to discover and exploit layers worth duplicating at training time.
The current model architectures we use have a fixed routing of residuals per layer, from the first to the last. I'm imagining replacing this with a matrix of routing weights[0] that determines how "strong" the connection is between each Transformer layer. We still evaluate each layer "in order", but now instead of just giving the layer the last layer's residuals, it gets the sum of all prior layers times their weight in the routing matrix. Recurrent connections (i.e. output of layer 9 to input of layer 3) could be handled by doing a second pass and using the first pass's recurrent residuals as inputs. You could then "loop" the model as many times as desired per token, or even have it do parallel decoding with each token communicating with the others while also recurring on itself.
You'd probably need some kind of normalization akin to what Deepseek did with Manifold Hyper Connections (mHC). Hell, mHC might also be useful in combination with this kind of layer routing, so the model could grow different recurrent loops for various bits of it's thought-space.
EDIT: if anyone uses it please call it "neuralese recurrence" just to scare the AI safety bros
[0] I'm not sure how you'd initialize these weights. Maybe each row/column is a narrow gaussian centered around the prior layer, with some random or constant weighting everywhere else?
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#25Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#26This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract th…
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
Yes, computing in latent space is a big thing now.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#27Earlier quoted context omitted.
Took me a short time to understand what you mean with "autoencoders on steroids", but I believe you mean they are autoencoders with an inverse bottleneck - an intermediate representation that isn't smaller, but that's much larger than the input space. Is my understanding of your comment correct?
Kind of. Autoencoders don’t need to have an embedding that’s smaller than the input. Their only requirement is that they compress information and thus create reconstruction loss. Typically however they are not trained this way because they don’t converge.. transformers do the same thing, but they can squeeze much more bits of information through one pass because the way they are designed. This holds true even for dec…
If the embedding isn’t smaller than the input, how is it compressing information? It might lose information in its mapping to the embedding space, but in my understanding, the definition of compression means it has to use less bits than the original to hold the same information. As such, the embedding space must be smaller.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#28Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#29This is interesting theoretically, but in practical terms it’s hard to apply.
RL is already hard. There are many things which can go wrong. You have all of the problems with regular LLM SFT, plus now you have a reward model which can be hacked or too hard. Or KL collapse because the outputs are repetitive. Or maybe your groups in GRPO aren’t producing advantages. Or the rollouts are OOD for your reward model. Or maybe you’re running the rollout at a different precision as the trained weights. Or maybe your importance sampling should be clipping when it’s not, or should be clipping at the token level rather than sequence level.
Maybe after reading the above you think that the above are not problems because smart people wouldn’t make those mistakes. Fair enough. But I would prefer RL people like myself who are not geniuses.
Now, this is adding another variable into the mix: choosing a single layer to train. If it doesn’t work is it because there’s a problem with your RL setup? Or did you just choose the wrong layer? Or maybe there’s no problem with your setup but you chose a suboptimal layer to train.
Also note that we already have LoRA, which is a more established method for low memory parameter updates.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#30This is interesting theoretically, but in practical terms it’s hard to apply. RL is already hard. There are many things which can go wrong. You have all of the problems with regular LLM SFT, plus now you have a reward model which can be hacked or too hard. Or KL collapse because the outputs are repetitive. Or maybe your groups in GRPO aren’t producing advantages. Or the rollouts are OOD for your reward model. Or mayb…
I guess the practical implication is probably just train on middle layers or increase the corresponding learning rate