Viewing profile — colah3

(Disclaimer: I work on interpretability at Anthropic.) I wanted to flag that this is an accessible blog post and that there's a link to the paper ( https://transformer-circuits.pub…

Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications. A few tho…

See https://transformer-circuits.pub/2022/toy_model/index.html#m... If you're new to this, I'd mostly just look at all the empirical examples. The slightly harder thing is to consi…

It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equi…

I guess I'll plug my hobby horse: The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions abo…

> True! I suppose I was thinking about a 'strong' form of linear representations, which is something like: features are represented by linear combinations of neurons that display t…

If you like symmetry, you might enjoy how symmetry falls out of circuit analysis of conv nets here: https://distill.pub/2020/circuits/equivariance/

> Circuits I find less compelling, since the analysis there feels very tied to the transformer architecture in specific, but what do I know. I don't think circuits is specific to t…

Since this post is based on my 2014 blog post ( https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a …

A few comments on this thread: Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range. I expect AI assistants could help q…

Yep, that’s right! If you want to be precise, there are “autoregressive transformers” and “bidirectional transformers”. Bidirectional is a lot more common in vision. In language mo…

Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the …

Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research resu…

> The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretabilit…

Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There ar…

Features correspond to vectors in activation space. So you can just do vector arithmetic! If you aren't familiar with thinking about features, you might find it helpful to look at …

I think the question is: by what mechanism does it adjust up the probability of the token "an"? Of course, the reason it has learned to do this is that it saw this in training data…

Just to be clear, the probability for "An" is high, just based on the prefix. You don't need to do beam search.

The planning is certainly performed by circuits which we learned during training. I'd expect that, just like in the multi-step planning example, there are lots of places where the …

I used the astronomer example earlier as the most simple, minimal version of something you might think of as a kind of microscopic form of "planning", but I think that at this poin…

"An astronomer" is two tokens, which is the relevant concern when people worry about this.

Yes, there are two kinds of evidence. Firstly, there is behavioral evidence. This is, to me, the less compelling kind. But it's important to understand. You are of course correct t…

Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy ( https://colah.github.io/ ). I think this post and its children have some important…

I'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similar…

I'm glad you've found it easy to follow! My best guess at the middle regime is that there are _empirical correlations between features_ due to the limited data. That is, even thoug…