Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mechanistic interpretability researchers applying causality theory to LLMs

cacm.acm.org

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#41

Post by blooalien »

Leonard_of_Q wrote: post #24
blooalien wrote: post #9

Earlier quoted context omitted.

> Do we see signs of reasoning or is it anthropomorphism? This is the part that so many folks just don't seem to understand (probably because it's been labeled as "thinking" or "reasoning" mode, and people assume that words have meaning). It's not reasoning or thought. It's spewing tokens pretending to "think", but it's actually just generating extra "context" to help the final answer be more coherent. The model isn'…

You're describing a process by which a 'thinking' entity uses cognition to refine a solution to a stated problem. That's a lot of words so usually we shorten this to 'reasoning'. Do LLMs 'think'? I 'think' they do in a way. I don't really know how I think myself but I know I do and therefore I am (thanks, Descartes). I have a somewhat better grasp of the way LLMs 'think'. They do so sequentially, building a chain of…

[flagged]

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#42

Post by smokel »

warumdarum wrote: post #18
dang wrote: post #61

[stub for offtopicness] [[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase , and we're trying for something more substantive here.]]

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and yo…

It's probably helpful in this discussion to make a difference between two definitions of reasoning:

1. phenomenal reasoning, requiring consciousness and subjective experience

2. functional reasoning, transforming premises into conclusions using logic

I think you are attacking this using definition 1, whereas the article is obviously aiming at a different type of reasoning, and trying to formalize what is actually going on. It seems to be a genuine effort.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#43

Post by CamperBob2 »

CrzyLngPwd wrote: post #14
dang wrote: post #61

[stub for offtopicness] [[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase , and we're trying for something more substantive here.]]

My toaster doesn't reason, and neither do the current clankers.

How'd your toaster do at IMO last year?

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#44

Post by CamperBob2 »

otabdeveloper4 wrote: post #13
dang wrote: post #61

[stub for offtopicness] [[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase , and we're trying for something more substantive here.]]

They don't reason.

What would change your mind?

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#45

Post by mexicojalisco »

rvba wrote: post #33
warumdarum wrote: post #18

Earlier quoted context omitted.

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and yo…

There is a streamer who plays Diablo 2 by listening to the AI advice and it is quite funny since it is pretty clear that most of the advice is an amalgamation of random, often incorrect advicem I wonder if it is the same for programming or not, but I vibe coded an android app just to see if I can and it just works. It required a lot of "build the code and correct the errors" pushing though. For example requested code…

As somebody who uses Claude heavily and heavily plays D2R it’s clear he wasn’t using Claude opus…… maybe Haiku or something. Opus isn’t as brain dead as what was being displayed

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#46

Post by throw310822 »

michaelchisari wrote: post #19
azakai wrote: post #12

Earlier quoted context omitted.

Yes, we do see signs of actual reasoning, see the papers linked in the article. (There are many others too.) Yes, we have a tendency to anthropomorphize, but (most) researchers are aware of this.

The papers linked in the article discuss the mechanical operations that simulate reasoning. Intelligence is data efficiency and I don't see a strong argument that reasoning can exist if it requires a world's worth of data. That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.

> a thing is not its simulation.

"The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circuits glowed red and gold as she picked simulated daisies, and hummed a simulated song."

(Stanislaw Lem, Cyberiad)

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#47

Post by crewindream »

alchemist1e9 wrote: post #28
warumdarum wrote: post #18

Earlier quoted context omitted.

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and yo…

It’s curious how they solve unsolved math problems without reasoning. Maybe I have a different definition of reasoning than you.

Jury is still out on this one.

This needs to be routine to be given asevidence…

…Unless you know exactly how the llm was trained and then how it was applied

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#48

Post by danbruc »

I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a hallucination. But those weights do not exist for the sake of it, they encode a lot of text the model has seen during training, and I would imagine this is what drives the reasoning. Can you evaluate the following polynomial ... will be related to To evaluate a polynomial ... seen in the training data. This is the level at which I would look for the reasoning, memorized patterns how to do specific things, maybe with some kind of placeholder variables for generalization. Ultimately such a structure would of course also be represented in the weights but I could imagine that this makes it unnecessary hard to understand. Or maybe not, maybe the learned patterns are so complex that they do not have a simple representation.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#49

Post by skybrian »

warumdarum wrote: post #18
dang wrote: post #61

[stub for offtopicness] [[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase , and we're trying for something more substantive here.]]

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and yo…

When a mathematician reads a hundred-year-old math paper, they are reproducing in their head the reasoning of someone who died long ago. That is, reasoning can be written down and replicated.

If that works, I think it's fair to say that LLM's are inanimate processes that can generate real reasoning. You can tell when you read it and it makes sense.

There are likely some kinds of reasoning that can't be written down, as well as other forms of understanding, but they also don't replicate nearly as easily.

Re: Mechanistic interpretability researchers applying causality theory to LLMs

#50

Post by michaelchisari »

throw310822 wrote: post #46
michaelchisari wrote: post #19

Earlier quoted context omitted.

The papers linked in the article discuss the mechanical operations that simulate reasoning. Intelligence is data efficiency and I don't see a strong argument that reasoning can exist if it requires a world's worth of data. That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.

> a thing is not its simulation. "The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circu…

"In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.

"Suarez Miranda,Viajes de varones prudentes, Libro IV,Cap. XLV, Lerida, 1658"

- On Exactitude in Science by Jorge Luis Borges

Post reply on HN