Interesting, reminds me of similar work Anthropic did on Claude 3 Sonnet [0]. [0] https://transformer-circuits.pub/2024/scaling-monosemanticit...
Someone mentioned that this took almost as much compute to train as the original model.
Extracting concepts from GPT-4
31–40 of 155 posts
Re: Extracting concepts from GPT-4
#32Earlier quoted context omitted.
LLM based AIs have lots of "features" which are kind of synonymous with "concepts" - these can be anything from `the concept of an apostrophe in the word don't`, to `"George Wash" is usually followed by "ington" in the context of early American History`. Inside of the LLMs neural network, these are mapped to some circuitry-in-software-esque paths. We don't really have a good way of understanding how these features ar…
hallucination solution?
Solving this problem would be a step on the way to debugging (and then resolving, or at least highlighting) hallucinations.
Re: Extracting concepts from GPT-4
#33Can someone ELI5 the significance of this? (okay maybe not 5, but in basic language)
High-level concepts stored inside the large models (diffusion models, transformers etc) are normally hard to separate from each other, and the model is more or less a black box. A lot of research is put into obtaining the insight into what model knows. This is another advancement in this direction; it allows for easy separation of the concepts.
This can be used to analyze the knowledge inside the model, and potentially modify (add, erase, change the importance) certain concepts without affecting unrelated ones. The precision achievable with the particular technique is always in question though, and some concepts are just too close to separate from each other, so it's probably not perfect.
Re: Extracting concepts from GPT-4
#34Can someone ELI5 the significance of this? (okay maybe not 5, but in basic language)
In general this is just copying work done by Anthropic, so there's nothing fundamentally new here.
What they have done here is to identify patterns internal to GPT-4 that correspond to specific identifiable concepts. The work was done my OpenAI's mostly dismantled safety team (it has the names of this teams recently departed co-leads Ilya & Jan Leike on it), so this is nominally being done for safety reasons to be able to boost or suppress specific concepts from being activated when the model is running, such as Anthropic's demonstration of boosting their models fixation on the Golden Gate bridge:
https://www.anthropic.com/news/golden-gate-claude
This kind of work would also seem to have potential functional uses as well as safety ones, given that it allow you to control the model in specific ways.
Re: Extracting concepts from GPT-4
#35This is interesting:
> Autoencoder family
> Note: Only 65536 features available. Activations shown on The Pile (uncopyrighted) instead of our internal training dataset.
So, the Pile is uncopyrighted, but the internal training dataset is copyrighted? Copyrighted by whom?
Huh?
Re: Extracting concepts from GPT-4
#36One feature I expect we'll get from this sort of research is identifying "hot spots" that are used during inference. Like virtual machines, these could be cached in whole or in part and used to both speed up the response time and reduce computation cycles needed.
Re: Extracting concepts from GPT-4
#37The worrying part is that first concept in the doc they show/found is "human imperfection". Hope this is just coincidence..
I think it's done on purpose, it's related to a very important point when understanding AI. Humans aren't perfect, AI is trained by humans, therefore...
I think the point of the doc is that "human imperfection" is very prominent concept in trained LLM..
Re: Extracting concepts from GPT-4
#38This is interesting: > Autoencoder family > Note: Only 65536 features available. Activations shown on The Pile (uncopyrighted) instead of our internal training dataset. So, the Pile is uncopyrighted, but the internal training dataset is copyrighted? Copyrighted by whom? Huh?
> Copyrighted by whom?
By people who would get angry if they could definitively prove their stuff was in OpenAI's training set.
Re: Extracting concepts from GPT-4
#39Exciting to see this so soon after Anthropic's "Mapping the Mind of a Large Language Model" (under 3 weeks). I find these efforts really exciting; it is still common to hear people say "we have no idea how LLMs / Deep Learning works", but that is really a gross generalization as stuff like this shows. Wonder if this was a bit rushed out in response to Anthropic's release (as well as the departure of Jan Leike from Op…
From the article: "We currently don't understand how to make sense of the neural activity within language models." "Unlike with most human creations, we don’t really understand the inner workings of neural networks." "The [..] networks are not well understood and cannot be easily decomposed into identifiable parts" "[..] the neural activations inside a language model activate with unpredictable patterns, seemingly re…
Not holding my breath for that hallucinated cure for cancer then.
Re: Extracting concepts from GPT-4
#40This is interesting: > Autoencoder family > Note: Only 65536 features available. Activations shown on The Pile (uncopyrighted) instead of our internal training dataset. So, the Pile is uncopyrighted, but the internal training dataset is copyrighted? Copyrighted by whom? Huh?
Basically everyone. You, and me, and Elon Musk, and EMPRESS, and my uncle who works for Nintendo. They're just hoping that AI training legally ignores copyright.