Beyond the Black Box: Interpretability of LLMs in Finance
1–10 of 15 posts
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#2Paper introduces AI explainability methods, mechanistic interpretation, and novel Finance-specific use cases. Using Sparse Autoencoders, we zoom into LLM internals and highlight Finance-related features. We provide examples of using interpretability methods to enhance sentiment scoring, detect model bias, and improve trading applications.
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#3[deleted]
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#4Paper introduces AI explainability methods, mechanistic interpretation, and novel Finance-specific use cases. Using Sparse Autoencoders, we zoom into LLM internals and highlight Finance-related features. We provide examples of using interpretability methods to enhance sentiment scoring, detect model bias, and improve trading applications.
[deleted]
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#5Cool stuff. I'm the CTO of Stargazr (stargazr.ai), a financial & operational AI for manufacturing companies; we started using transformers to process financial data in 2020, a bit before the GPT boom.
In our experience, things beyond very constrained function calling opens the door to explainability problems. We moved away from "based on the embeddings of this P&L, you should do X" towards "I called a function to generate your P&L, which is in this table; based on this you could think of applying these actions".
It's a loss in terms of semantics (the embeddings could pack more granular P&L observations over time) but much better in terms of explainability. I see other finance AIs such as SAP Joule also going in the same direction.
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#6Cool stuff. I'm the CTO of Stargazr (stargazr.ai), a financial & operational AI for manufacturing companies; we started using transformers to process financial data in 2020, a bit before the GPT boom. In our experience, things beyond very constrained function calling opens the door to explainability problems. We moved away from "based on the embeddings of this P&L, you should do X" towards "I called a function to gen…
Thank you. Agreed, we are exploring different ways to apply these interpretability methods to a wide range of transformer based methods, not just decoder based generative applications.
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#7I’m still waiting for somebody to explain to me how a model with a million+ parameters can ever be interpretable in a useful way. You can’t actually understand the model state, so you’re just making very coarse statistical associations between some parameters and some kinds of responses. Or relying on another AI (itself not interpretable) to do your interpretation for you. What am I missing?
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#8I’m still waiting for somebody to explain to me how a model with a million+ parameters can ever be interpretable in a useful way. You can’t actually understand the model state, so you’re just making very coarse statistical associations between some parameters and some kinds of responses. Or relying on another AI (itself not interpretable) to do your interpretation for you. What am I missing?
Even a large model has to behave fairly predictably to be useful; it's not totally random, is it? The same thing applies to humans.
Interpretability can mean several things. Are you familiar with things like this? https://distill.pub/2018/building-blocks/
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#9Thanks Ariye. What does group risk think about this paper?
I imagine these metrics would be good to include in the MI but are you confident that the methods being proposed are adequate to convince regulators on both sides of the Atlantic?
Re: Beyond the Black Box: Interpretability of LLMs in Finance
#10I’m still waiting for somebody to explain to me how a model with a million+ parameters can ever be interpretable in a useful way. You can’t actually understand the model state, so you’re just making very coarse statistical associations between some parameters and some kinds of responses. Or relying on another AI (itself not interpretable) to do your interpretation for you. What am I missing?
Our paper provides evidence of features in Finance but I would suggest reading seminal papers from Anthropic https://www.anthropic.com/news/golden-gate-claude and https://transformer-circuits.pub/2024/scaling-monosemanticit...
Monosemantic behavior is key in our research.