Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Steerling-8B, a language model that can explain any token it generates

guidelabs.ai

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#31

Post by vintagedave »

adebayoj wrote: post #18
ottah wrote: post #10

It's a neat party trick, but explainability it's not solution to any AI safety issue I care about. It's a distraction from real problems, which is everything else around the model. The inflexible bureaucratic systems that make it hard to exercise rights and deflect accountability.

op here. Important point, but I disagree. We see explainability/interpretability as a CORE need for AI safety. We believe you can't align/audit/debug/fix a system that you don't understand. Just to give you some answers for what we can do: 1) We can find the training data that is causing a model to output toxic/unwanted text and correct it. 2) We know what high level concepts the model is relying on for any group of…

This is fantastic to read. LLMs feel like black boxes and for the large ones especially I have a sense they genuinely form concepts. Yet the internals were opaque. I remember reading how LLMs cannot explain their own behaviour when asked.

I feel this would give insight into all that including the degree of true conceptualisation. I’m curious if this can demonstrate what else the model is aware of when answering, too.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#32

Post by yorwba »

7777777phil wrote: post #21

If this decomposition actually holds, it's the first model where you could show a regulator why it produced a given output.

I doubt that a regulator would be satisfied by the kinds of explanations this provides and the interventions it enables.

Suppose somebody put an LLM in charge of an industrial control system and it increased the temperature so much that it caused an accident. The input feature attribution analysis shows that the model was strongly influenced by the tokens describing the temperature control mechanism, concept attribution shows that it output tokens related to temperature, industrial processes and LLM tool-call syntax.

The operator proposes to fix this by rewriting the description and downweighting the temperature concept in the output, and a simulation shows that with these changes the model doesn't make the same decisions in this situation anymore. Should the regulator accept this explanation as sufficient to establish that the system is now safe?

If the controller has just a few parameters and responds approximately linearly to changes in its inputs, you can in principle guarantee that it'll stay within a safe zone. But LLMs have a huge number of parameters and by design highly nonlinear behavior. A simple explanation is unlikely to reflect model behavior accurately enough that you can trust its predictions to hold in arbitrary situations.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#33

Post by adebayoj »

umairnadeem123 wrote: post #11

[dead]

Good point. Historically, people have thought that there is a interpretability vs quality/performance tax. This is not true; at least not in this case.

Here are a bunch of questions you can answer without any quality degradation with interpretable models: 1) what part of the input context led to the output chunk that the model generated? 2) what part of the training data led to the output chunk?

In this case, we go more invasive, and actually constrain the model to also use human understandable concepts in its representations. You might think this leads to quality trade-offs. However, if you allow for the model to discover its own concepts as well (as long as they are not duplicates of the concepts you provided it), you don't see huge degradation.

I agree with the other commenters that this now gives us a huge boost in debugging the model.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#34

Post by adebayoj »

luulinh90s wrote: post #13
umairnadeem123 wrote: post #11

[dead]

in the "Performance" section of the post: https://www.guidelabs.ai/post/steerling-8b-base-model-releas... , the authors show the model lags behind llama 8b but worth noting that llama 8b trained on > 2x more computes (see the FLOPs axis)

Thanks for pointing this out. LLama 3 8B was trained on ~15T tokens. The Qwen models on 15-18T tokens as well. We trained on 1.35T tokens, and are within shot of these models on benchmarks. We expect to, at the very minimum, match these models' performance when we scale our token budget.

One side effect that we are excited about is that interpretable model training might make for a data efficient training process.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#35

Post by adebayoj »

vintagedave wrote: post #31
adebayoj wrote: post #18

Earlier quoted context omitted.

op here. Important point, but I disagree. We see explainability/interpretability as a CORE need for AI safety. We believe you can't align/audit/debug/fix a system that you don't understand. Just to give you some answers for what we can do: 1) We can find the training data that is causing a model to output toxic/unwanted text and correct it. 2) We know what high level concepts the model is relying on for any group of…

This is fantastic to read. LLMs feel like black boxes and for the large ones especially I have a sense they genuinely form concepts. Yet the internals were opaque. I remember reading how LLMs cannot explain their own behaviour when asked. I feel this would give insight into all that including the degree of true conceptualisation. I’m curious if this can demonstrate what else the model is aware of when answering, too.

Our decomposition allows us to answer question like: for 84 percent of the model's representation, we know it is relying on this concept to give an answer.

We can also trace its behavior to the training data that led to it, so that can show us where some of these concepts are formed from.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#36

Post by adebayoj »

great_psy wrote: post #16
rickydroll wrote: post #12

Earlier quoted context omitted.

Your question should be "Can it drill down to show the paragraphs or sentences that influence the answer?" I believe that the plagiarism complaint about llm models comes from the assumption that there is a one-to-one relationship between training and answers. I think the real and delightfully messier situation is that there is a many-to-one relationship.

The example on the website shows one to many as well: Wikipedia, axive article, etc along with a ratio how much it influences the chunk of the answer.

Exactly! We will have a future post that shows this more granularly over the coming weeks. Here is a post we wrote on how this works at smaller scale: https://www.guidelabs.ai/post/prism/

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#38

Post by potato-peeler »

Looks very interesting. Is there a published paper/article on your algorithm? Would like to take a dab at implementing this on my own.

I could find this [0], but not sure if that represents the entire system? (Apologies, I am not that well versed in ML)

[0] - https://www.guidelabs.ai/post/scaling-interpretable-models-8...

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#39

Post by andy12_ »

This seems really interesting. While Anthropic tried to use dictionary learning over an existing model to try to extract concepts, this almost feels like training the model alongside the dictionary itself (or rather, the model and the dictionary are intertwined).
Post reply on HN