Makes sense - This is very similar to fine tuning a down stream task in encoder-decoder architecture (~Bert style)
Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
11–20 of 44 posts
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#12Earlier quoted context omitted.
Now that’s interesting.. what exactly distinguishes latent representations and the manifold? IMHO, those are the same, and you’re constructing a piecewise function of the manifold itself. Decoders also produce manifolds much in the same way, with the distinction being that the encoder isn’t learned but static after initialisation. So fundamentally it is still DOING the same operation.
The latent representations of the data are like points on a surface. That surface is the manifold. We don't typically have the full manifold and can only sample points from it by embedding data into it. Worth noting a different manifold "exists" after each transformation (e.g. layer). You only sample from the same manifold when you apply the same transformation(s).
Also worth noting that in reality manifolds will be "spiky" in very high dimension, so the idea of a "surface" is best understood through patterns of distance between samples in embedding space and way they collapse in low D.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#13It's interesting that it's the middle layers of the Transformer that are affected most by RL post-training, but it perhaps makes some intuitive sense given that RL is being used to shape high level planning-type direction of the output.
It seems that the input layers to a Transformer are necessarily going to be doing the most low level work of syntax -> semantic augmentation starting with things like tagging parts of speech etc. Similarly the output layers are by necessity going to be concerned with mapping high level representations back into surface level word sequence form. This leaves the middle layers to do the work of first recognizing deep enough patterns to support good quality prediction, then do the high level predication itself which is what RL is typically going to be trying to shape.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#14I'm reminded of this dude who was sitting at or near the top of some kaggle leaderboard simply[0] splicing together some duplicated middle layers and applying a bit of fine tuning
[0] not simply
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#15I still can't believe that LLM encoders aren't unsupervised learned.
So much left on the table
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#16The authors have some inconsistencies with training token length…
Most errors are probably responses that didn’t finish before their 3K token limit. They’ve measured how well RL is able to shorten the response to their limit.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#17If you think about it for some time then you’ll come to realise transformers are autoencoders on steroids. A small input space is expanded onto a big manifold and contracted again. Now, suppose you want to impose a function to regulate the output of an autoencoder. It’s actually pretty obvious that you need exactly one layer to do so… f(manifold).
What you're suggesting seems to go implausibly far beyond what the paper says. RL post-training alters the parameters of the transformer, while your f(manifold) idea seems to suggest that a new layer on top would suffice, no need to alter the transformer itself at all. It would be extremely handy if that were so, but I'm guessing it isn't, or it would be the prevailing approach.
The manifold is in the middle (“small input space is expanded onto a big manifold and contracted again”) so f(manifold) would need to be in the middle too.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#18This result feels very intuitive. The early layers of a transformer can be thought of as understanding surface level things like syntax, how tokens group, which groups are entities and how to disambiguate them, etc. The last layers are in a sense decoding ideas into a selection of words, ensuring the grammar makes sense, that the text flows and is structured correctly, etc. The middle layers are where the abstract th…
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it.
https://dnhkng.github.io/posts/rys/
Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#19If you think about it for some time then you’ll come to realise transformers are autoencoders on steroids. A small input space is expanded onto a big manifold and contracted again. Now, suppose you want to impose a function to regulate the output of an autoencoder. It’s actually pretty obvious that you need exactly one layer to do so… f(manifold).
Everything can be represented as f(), a full scale SotA transformer model is also just f(context). That does not mean one layer is sufficient. It all depends on the level of expressivity required by this f to be a good model.
Re: Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
#20I'm reminded of this dude who was sitting at or near the top of some kaggle leaderboard simply[0] splicing together some duplicated middle layers and applying a bit of fine tuning [0] not simply
Was it this guy? https://news.ycombinator.com/item?id=47322887