> What color is the planet fourth from the sun?
Are they trying to show internal consistency even when the produced answer is wrong?
A global workspace in language models
171–180 of 218 posts
Re: A global workspace in language models
#172> What color is the planet fourth from the sun? Are they trying to show internal consistency even when the produced answer is wrong?
I dont understand what your comment is referring to
Re: A global workspace in language models
#173Earlier quoted context omitted.
It certainly holds true for humans. The brain stores relational information in a sequential pattern that is not automatically reversible. One of the best examples is the alphabet. Everyone learns it in school, so the pattern A->B->C->... is trivial to recite for most people. Now, if I gave you a random letter to start with and asked you to to recite the remaining letters until Z, you'll probably find it is still pret…
Is the alphabet really the same though? I don't feel like I recall it one letter at a time, as individual facts linking A to B, then B to C, etc, but more as a sound or a phrase. Not unlike recalling a melody. It just seems very different from figuring out what band someone is describing.
Not that I'm too sure about how this applies (or whether it applies at all) to other people as well, but for me personally it's easier to recall information about the band if I'm being told the band's name instead of being told "well, they are known for coloured ties". So, there is certainly some kind of effect described in action.
Now, about alphabet: again, I think it's only me, but when I try to recall it backwards, I can't do that easily. I mean, I can recall it backwards, but I need more time to do that. It's harder. I'm not sure if it's because A links to B, B to C, then C to D and not backwards, or maybe just because in school you learn alphabet from A to Z and not from Z to a - so you're kind of trained to recall it A-->Z way - but it's certainly harder for me.
At the end of the day, though, I think that everyone thinks differently. Everyone is having different internal representations for concepts (such as alphabet), so it’s not surprising that this effect may work differently for different people, or not work at all.
Re: A global workspace in language models
#174[flagged]
Re: A global workspace in language models
#175This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning…
I'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.
They're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.
Re: A global workspace in language models
#176Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts:
> The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.
This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?
Other misc observations:
• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?
• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.
• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?
• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?
• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?
Re: A global workspace in language models
#177Earlier quoted context omitted.
Found it: https://news.ycombinator.com/item?id=47500709 Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/ tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed…
Nah, it's a cool blog post especially as it was real AI research done at home (albeit with a ridiculously expensive PC), but Anthropic and other labs have been investigating this kind of thing for years. Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as bein…
Author here:
Yeah, the encoder and decoder stuff is explicit, but the internal structure in generated during training. I don't think the big labs were doing this back when I did the research; no one was back in '24.
I just didn't get round to publishing for years, because I have a day job.
By the way, it still works! I tested it earlier this year on Qen3.6 and you still see improvements, so either a) no one actually paid attention, or b) it has more room to scale.
Re: A global workspace in language models
#178Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts: > The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons. This sounds suspiciously saying the models must follow the strong Sapir-Whorf hy…
They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.
Re: A global workspace in language models
#179Earlier quoted context omitted.
> I’d prefer a more direct claim in a paper This is not written to be just a paper. The target audience include media and online forums, and then maybe academia. Edit: typo
I’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.
Well I was agreeing with you. Skimmed the paper as well.
Re: A global workspace in language models
#180Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts: > The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons. This sounds suspiciously saying the models must follow the strong Sapir-Whorf hy…
They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same charact…
My interpretation of prior mechanistic interpretability research is that LLMs transform concepts in non-verbal spaces as well, which is why that was a surprising statement. For example showing how they do arithmetic on line lengths by rotating helical manifolds.
And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.
Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.