> We (the authors of this website) have at times sought insight into the inner workings of an LLM by asking it “why did you just do that?”
> But the LLM can’t tell us. It’s not a person. It doesn’t have the metacognitive abilities necessary to reflect on its past actions and report the motivations underlying them*.
> With no clue why it did whatever it just did, the LLM is forced to guess wildly at a plausible explanation, like the ill-fated Leonard Shelby in Christopher Nolan's film Memento.
> And we, gullible humans that we are, often believe its bullshit.
---
I am almost convinced that we ourselves are a narrator riding along inside an animal's mind, trying desperately to put together explanations for our actions, mostly just to convince others, just as though an LLM were running on our own senses trying to portray some deep semblance of consciousness. I don't think we'll find a super smart AI, we'll just realize we were not very sophisticated all along. The power of speech for information and culture transfer, writing, inspiration, and coordination is just awe inspiring, evolutionary speaking, so once we could talk we had to, because the "better" talker almost always won. It's an arms race.