Earlier quoted context omitted.
LLMs aren't monolingual so you might want to expand that beyond English. Consider how multilingual people think. Also, apparently it's pretty common for people to think in words and have an internal monologue. I hadn't realized this was a thing until recently but it seems many people don't think abstractly as you've described.
There have been psychological experiments that have shown what OP experiences. In this particular experiment subjects were asked to make a series of arbitrary choices between 2 items, while being hooked up to some brain scanning equipment. Subjects were asked to think about it for 5 seconds before making the choice. The experiment showed that scientists were able to predict the choice before the subjects consciously…
I wonder if they could plot the mind-changing process resulting from that sort of argument. If that is what even happens.