I’ve struggled with this line of thought. In a strict sense, it’s true; an AI model doesn’t actually know why it fired the way it did. But I don’t know how practically useful that observation is because, using that strict definition of “know”, humans also do not know why they do what they do.
If I asked you why you posted this comment, I suspect you’d be able to articulate a good reason why from your memories. But the firing of your subconscious and the entirety of your lived experience that molded the state in your brain that lead to you pressing that reply button and writing those exact words are inaccessible to you.
It reminds me of the timeline exercise in SRE work. I always start with a timeline and plunk pagers, metrics, slack messages, etc onto that timeline first. Then I go to individual people to get their understanding of the incident and get that into the timeline. People’s understanding of the incident and why they did what they did rarely matches the timeline.