Seems like this might be used as an alternative to embeddings for creating semantic search. Run a document collection through this to get the activations, run the prompt through the same thing and use the sparse autoencoder to identify which documents has the most activation of features in common with features that the prompt activates. You don't have to know what the features are, but if the prompt includes, say a rhetorical question, it may be able to find documents that also includes rhetorical questions. Not sure how well this would work, but it seems like something that could be interesting to try.
Extracting concepts from GPT-4
131–140 of 155 posts
Re: Extracting concepts from GPT-4
#132Earlier quoted context omitted.
From the article: "We currently don't understand how to make sense of the neural activity within language models." "Unlike with most human creations, we don’t really understand the inner workings of neural networks." "The [..] networks are not well understood and cannot be easily decomposed into identifiable parts" "[..] the neural activations inside a language model activate with unpredictable patterns, seemingly re…
I read this as "we have not built up tools / math to understand neural networks as they are new and exciting" and not as "neural networks are magical and complex and not understandable because we are meddling with something we cannot control". A good example would be planes - it took a long while to develop mathematical models that could be used to model behavior. Meanwhile practical experimentation developed decent…
I think you have to make a distinction between transformers and neural networks in general, maybe also between training and inference.
Many/most types of neural network such as CNNs are well understood since there is a simple flow of information. e.g. In a CNN you've got a hierarchy of feature detectors (convolutional layers) with a few linear classifier layers on top. Feature detectors are just learning decision surfaces to isolate features (useful to higher layers), and at inference time the CNN is just detecting these hierarchical features than classifying the image based on combinations of these features. Simple.
Transformers seem qualitatively different in terms of complexity of operation, not least because it seems we still don't even know exactly what they are learning. Sure, they are learning to predict next word, but just like the CNN whose output classification is based on features learnt by earlier layers, the output words predicted by a transformer are based on some sort or world model/derived rules learned by earlier layers of the transformer, which we don't fully understand.
Not only don't we know exactly what transformers are learning internally (although recent interpretability work gives us a glimpse of some of the sorts of things they are learning), but also the way data moves through them is partially learnt rather than proscribed by the architecture. We have attention heads utilizing learnt lookup keys to find data at arbitrary positions in the context, and then able to copy portions of that data to other positions. Attention heads learn to coordinate to work in unison in ways not specified by the architecture, such as the "induction heads" (consecutive attention head pairs) identified by Anthropic that seem to be one of the work horses of how transformers are working and copying data around.
Additionally, there are multiple types of data learnt by a transformer, from declarative knowledge ("facts") that seem to mostly be learnt by the linear layers to the language/thought rules learnt by the attention mechanism that then affect the flow of data through the model, as discussed above.
So, it's not that we don't know how neural networks work (and of course at one level they all work the same - to minimize errors), but more specifically that we don't fully know how transformer-based LLMs work since their operation is a lot more dynamic and data dependent than most other architectures, and the complexity of what they are learning far higher.
Re: Extracting concepts from GPT-4
#133Exciting to see this so soon after Anthropic's "Mapping the Mind of a Large Language Model" (under 3 weeks). I find these efforts really exciting; it is still common to hear people say "we have no idea how LLMs / Deep Learning works", but that is really a gross generalization as stuff like this shows. Wonder if this was a bit rushed out in response to Anthropic's release (as well as the departure of Jan Leike from Op…
[deleted]
Re: Extracting concepts from GPT-4
#134Sad to note that the team that built this is now completely defunct.
Re: Extracting concepts from GPT-4
#135Earlier quoted context omitted.
Physicists would probably argue that the system might be understood but that we don’t have the model for it yet. Many natural phenomena look chaotic at best without a model. Once you have a model things fall into place and everything starts looking orderly. Maybe it cannot be reduced. But maybe we are just observing the peripherals without understanding the inner workings.
If I can speak in aphorisms, Creation is downhill, analysis is uphill. Profound ideas often seem simple once understood.
Well put.
In other words: simplicity is a hallmark of understanding.
Re: Extracting concepts from GPT-4
#136Earlier quoted context omitted.
We know exactly what the system is capable of doing. It’s capable of outputting tokens which can then be converted into text.
Which is so broad as to be unhelpful. We also know that petroleum mixed with air may be combusted to release energy; we needed to characterise this much better in order for the motor car to be distinguishable from a fuel-air bomb.
And that's exactly my point. Regulating the underlying tech is utterly pointless in this case - it's utterly harmless by itself.
Re: Extracting concepts from GPT-4
#137Interesting, reminds me of similar work Anthropic did on Claude 3 Sonnet [0]. [0] https://transformer-circuits.pub/2024/scaling-monosemanticit...
The methods are the same, this is just OpenAI applying Anthropic's research to their own model.
The biggest thing I noticed comparing the two was that OpenAI's method really approached (and appears to have effectively mitigated) the dead latents problem with a clever weight initialization and an "auxiliary loss" which (I think) explicitly penalizes dead latents. The TopK activation function is the other main difference I spot between the two.
Now, on the flip side, the Anthropic effort goes much further than the OpenAI one in terms of actually doing something interesting with the outputs of all this. Feature steering and the feature UMAP are both extremely cool, and to my knowledge the OpenAI team stopped short of efforts like that in their paper.
Re: Extracting concepts from GPT-4
#138Earlier quoted context omitted.
From the article: "We currently don't understand how to make sense of the neural activity within language models." "Unlike with most human creations, we don’t really understand the inner workings of neural networks." "The [..] networks are not well understood and cannot be easily decomposed into identifiable parts" "[..] the neural activations inside a language model activate with unpredictable patterns, seemingly re…
Could there also be a “legal hedging” reason for why you would release a paper like this? By reaffirming that “we don’t know how this works, nobody does” it’s easier to avoid being charged with copyright infringement from various actors/data sources that have sued them.
If you know how it works, you can make it better,faster,cheaper.
Without the 300k starting salaries. I imagine that is a stronger incentive.
It's the users of the LLMs that want to launder repsonsibility behind "computer said no".
Re: Extracting concepts from GPT-4
#139"akin to the small set of concepts a person might have in mind when reasoning about a situation"
That does not necessarily mean only a small set of neurons in our brains are engaged.
It could be that we use the whole set or a large portion, albeit in a more efficient way!
Re: Extracting concepts from GPT-4
#140Sad to note that the team that built this is now completely defunct.
what do you mean?