Good work! I wonder if meta-learning can play a better role here compared to heuristics or hindsight. MAML requires hessians, but first-order MAML or Reptile variants could help apply layer-wise adjustments to learning rates.
Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
31–40 of 44 posts
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#32This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract th…
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#33Earlier quoted context omitted.
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
there certainly are experiments in keeping the reasoning in latent space.
i dont think this is quite the same though, since you arent picking tokens for the chain of thought. inatead, its staying on trying to pick the immediate next token.
as an alternative, maybe you could stack these to produce most likely token lists instead by stacking these?
but i think youd end up with the similar blurriness that llm video generators get where theyre returning an average of all the likely combinations rather than collapsing that wave function
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#34This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract th…
I've often thought about how rich a machine language could be that communicates machine to machine on an interface that is really really close to those middle layers. I imagine a standardize meta interface that each model 'grows' a connection to with RL.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#35It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that RL is being used to shape high level planning-type direction of the output. It seems that the input layers to a Transformer are necessarily going to be doing the most low level work of syntax -> semantic augmentation starting with things like tagging parts of…
this is explicitly why qwen opted to use recurrent layers as the middle layers of their hybrid model.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#36Earlier quoted context omitted.
What you're suggesting seems to go implausibly far beyond what the paper says. RL post-training alters the parameters of the transformer, while your f(manifold) idea seems to suggest that a new layer on top would suffice, no need to alter the transformer itself at all. It would be extremely handy if that were so, but I'm guessing it isn't, or it would be the prevailing approach.
The manifold is in the middle (“small input space is expanded onto a big manifold and contracted again”) so f(manifold) would need to be in the middle too.
The problem really is one has to find out what constitutes the embedding and output if you want to apply f(manifold) but as others have pointed out that’s out of scope for this paper. My only insight here is that it is not surprising at all that one layer, or even a single function pass, suffices to get the desired reinforcement. I don’t actually know if there’s any research that quantifies the “manifold threshold” of each layer to try out this approach but it would be interesting for sure
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#37Earlier quoted context omitted.
Kind of. Autoencoders don’t need to have an embedding that’s smaller than the input. Their only requirement is that they compress information and thus create reconstruction loss. Typically however they are not trained this way because they don’t converge.. transformers do the same thing, but they can squeeze much more bits of information through one pass because the way they are designed. This holds true even for dec…
If the embedding isn’t smaller than the input, how is it compressing information? It might lose information in its mapping to the embedding space, but in my understanding, the definition of compression means it has to use less bits than the original to hold the same information. As such, the embedding space must be smaller.
You’re thinking information content in the neurons but not distributions. D(world of text) is much much much larger than D(transformer). Just because you’re passing information through a smaller channel doesn’t mean the original distribution necessarily isn’t smaller. Tokenisers are the numbers but then you have V^n where V is your vocabulary and n is the sequence length as a subset of the whole distribution of possible text. Now you pass that through something with sufficient capacity, which compresses the input distribution (you loose bits), and then recreate from your lossy distribution. AutoEncoders are not trained this way because they simply don’t converge if you try to pass smaller input to produce larger embeddings, but they’re doing the same thing.
Another way is to think of it is that LLM layers act like binary classifiers via soft max for the input distribution (there were a slew of papers comparing LLMs to SVMs). Essentially during your training you decide what part of the input distribution is going to be part of the embedding.
In my humble opinion, while the training diverges, these models produce the same manifold induced by the original distribution.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#38This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract th…
I've often thought about how rich a machine language could be that communicates machine to machine on an interface that is really really close to those middle layers. I imagine a standardize meta interface that each model 'grows' a connection to with RL.
This idea is called neuralese and the labs are reluctant to do it because of the interpretability and control issues it poses.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#39Earlier quoted context omitted.
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
Is that in effect doing the same as “chain of thought” or “think through your steps” aka “Reasoning Models”?
It seems related. The primary benefit of reasoning tokens seems to be just giving the model more chances to loop through all its layers. There were experiments at one point with "pause tokens" that were meaningless but seemed to improve performance. Reasoning is a lot more interpretable though.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#40I'm reminded of this dude who was sitting at or near the top of some kaggle leaderboard simply[0] splicing together some duplicated middle layers and applying a bit of fine tuning [0] not simply
I'm wondering if the big problem is just the lack of recurrent connections in the standard Transformer design, and selective layer duplication is just a weird way to fix the same problem. I have to wonder if it would be possible to deliberately architecture a model to discover and exploit layers worth duplicating at training time. The current model architectures we use have a fixed routing of residuals per layer, fro…
Moe style router?