If this decomposition actually holds, it's the first model where you could show a regulator why it produced a given output.
Show HN: Steerling-8B, a language model that can explain any token it generates
21–30 of 104 posts
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#22This is very interesting. I don't see much discussion of interpretability in day to the day discourse of AI builders. I wonder if everyone assumes it to either be solved, or to be too out of reach to bother stopping and thinking about.
Most interpretability techniques haven't yet to be shown to be useful for everyday model pipelines. However, the field is working hard to change this.
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#23Earlier quoted context omitted.
It makes the black box slightly more transparent. Knowing more in this regard allows us to be more precise—you go from prompt tweak witchcraft and divination to more of possible science and precise method.
Can this method be extended to go down to the sentence level ? In the example it shows how much of the reason for an answer is due to data from Wikipedia. Can it drill down to show paragraph or sentence level that influences the answer ?
Great questions. We have several posts in the works that will drill down more into these things. The model was actually designed to answer these questions for any sentence (or group of tokens it generates).
It can tell you which specific text (chunk) in the training data that led to the output the model generated. We plan to show more concrete demos of this capability over the coming weeks.
It can tell you where in the model's representation it learned about science, art, religion etc. And you can trace all of these to either to input context, training data, or model's representations.
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#24Earlier quoted context omitted.
op here, I mostly agree with your comment! However, our model does more than this. For any chunk the model generates, it can answer: which concept, in the model's representations, was responsible for that token(s). In fact, we can answer the question: what training data caused the model to be generated too! We force this to be a constraint as part of the architecture and the loss function for our you train the model.…
All of the examples on the linked page seem to be "good" outputs. Attribution sounds most useful to me in cases where an LLM produces the typical kind of garbage response: wrong information in the training data, hallucinations, sycophancy, over-eagerly pattern matching to unasked but similar, well-known questions. Can you give an example of a bad output, and show what the attribution tells us?
You got it exactly right. Guilty as charged. Over the coming weeks, we will be showcasing exactly how you can debug all of these examples.
I agree that attribution is most useful for debugging and auditing. This is a prime usecase for us. We have a post with exciting results lined up to do this. Should be out in a week, we wanted to even just get the initial model out :)
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#25If this decomposition actually holds, it's the first model where you could show a regulator why it produced a given output.
It does :) We constrained the model to do exactly this during training: https://www.guidelabs.ai/post/scaling-interpretable-models-8....
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#26Can you use this to decrease hallucinations?
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#27If this decomposition actually holds, it's the first model where you could show a regulator why it produced a given output.
It does :) We constrained the model to do exactly this during training: https://www.guidelabs.ai/post/scaling-interpretable-models-8... .
thanks for getting back to me, very cool if true :) I have been asked about this many times when talking LLM use cases at enterprise level. Would love to run som tests, pleas shoot me a message to the email in my profile.
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#28Either I'm missing something or this is way overstated. Steerling appears to be just a discrete diffusion model where the final hidden states are passed through a sparse autoencoder (a common interpretability layer) before the LM head. They also use a loss that aligns the SAE'S activations with labelled concepts? However, this is an example of "The Most Forbidden Technique" [1], and could make the model appear interp…
You are missing a few things, but you got some things right.
1) The is not an SAE in the way you think. It is a combination of a supervised + unsupervised layer that is constrained. An SAE is typically completely unsupervised, and applied post hoc. Here, we supervise 33k of the concepts with concepts that we carefully curated. We then have an unsupervised component (similar to a topk SAE) that we constrain to be independent from the supervised concepts. We don't do any of this post hoc by the way; this is a key constraint. I"ll get back to this. We train that unsupervised layer along with the model during pre-training.
2) Are the concepts or features causally influential for the output? We directly use the combination of the concepts for the lm head, which is a linear transform (with activation), so we can tell you, in closed form, the effect of ANY concept on the output logit for any token (or group of tokens) generated. It is not just causally related, it is constrained to do so.
3) Other points: we also make it so that you can trace the model outputs to the training data. This is an underrated interpretability knob. You know where, and what data, caused your model to learn a particular feature.
This is already a long comment, but I want to close on why our approach sidesteps all the issues with SAEs. - If you train an SAE twice, on the same data + model, you'll get two different feature(s). - In fact, there is no reason, why the model should pick features that are causally influential for the output. - ALL of these problems stem from the fact that the SAE is trained AFTER you already trained your model. Training from scratch AND with supervision allows you to sidestep these issues, and even learn more disentangled representations.
Happy to more concretely justify the above. Great observations!
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#29Now this is something which is very interesting to see and might be the answer to the explainability issue with LLMs, which can unlock a lot more use-cases that are off limits. We'll see.
Thanks, it is certainly a first step.
Re: Show HN: Steerling-8B, a language model that can explain any token it generates
#30Earlier quoted context omitted.
It does :) We constrained the model to do exactly this during training: https://www.guidelabs.ai/post/scaling-interpretable-models-8... .
thanks for getting back to me, very cool if true :) I have been asked about this many times when talking LLM use cases at enterprise level. Would love to run som tests, pleas shoot me a message to the email in my profile.
sounds great! Will follow up via email.