Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mechanistic interpretability researchers applying causality theory to LLMs

cacm.acm.org

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#71

Post by hackinthebochs »

emp17344 wrote: post #30
alchemist1e9 wrote: post #28

Earlier quoted context omitted.

It’s curious how they solve unsolved math problems without reasoning. Maybe I have a different definition of reasoning than you.

Guess what? SAT solvers have also solved unsolved math problems. Do you believe they are “reasoning”?

SAT solvers are programs designed such that their execution corresponds to the reasoning process of satisfying some given constraints. But they do not contain the reasoning process, rather they embody it.

LLMs are different in that they operate on semantic features of program state. Embedding vectors assign semantic features to syntactical structures of the vector space. Operations on these syntactical structures allow the LLM to engage with semantic features of program state directly. Here the reasoning process is contained within as an object of manipulation. An LLM sensitive to the semantic features of the input sequence and that examines the logically permissible moves to derive a new sequence closer to the intended sequence (some statement to prove) just is engaging in reasoning.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#74

Post by hsb3 »

calf wrote: post #17

One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights. So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be t…

You dont have to understand chemistry to be a good cook tho.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#76

Post by azakai »

dominotw wrote: post #57

>“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.” what is the basis for this optimism ?

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#78

Post by dominotw »

azakai wrote: post #76
dominotw wrote: post #57

>“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.” what is the basis for this optimism ?

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

no they havent . success so far is totally meaningless and doesn't imply any sort of upward slope .

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#79

Post by azakai »

dominotw wrote: post #78
azakai wrote: post #76

Earlier quoted context omitted.

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

no they havent . success so far is totally meaningless and doesn't imply any sort of upward slope .

The researchers in the field disagree with you. Look at conferences like NeurIPS and ICLR to see a steady stream of incremental progress in this area.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#80

Post by dpark »

danbruc wrote: post #48

I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a h…

> In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights.

What is an output token distribution except a set of weights?

Post reply on HN