Bag of words, have mercy on us

> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?

So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."

That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.

> Language and society constrains the way we use words, but when you speak, are you "predicting"?

Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:

* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.

* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.

* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.

The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.

[0] https://www.newdualism.org/papers/E.Feser/Feser-acpq_2013.pd...

If you read the above link you will see that they spell out 3 problems with our understanding of thought:

Consciousness, intentionality, and rationality.

Of these, I believe prediction is only necessary for intentionality, but it does have some roles to play in consciousness and rationality.

I take a offence in the idea I’m “religiously downplaying LLMs”. I pay top dollar for access to the best models because I want the capabilities to be good / better. Just because I’m documenting my experience it doesn’t mean I have an Anti-ai agenda ? I pay because I find LLMs to be useful. Just not in the way suggested by the marketing teams.

I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?

I use all of the things you talk about extremely frequently and again, there is no “thinking” or consideration on display that suggests these things work like us, else why would we be having this conversation if they were ?

> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.

Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.

As a consequence of my profession, I understand how LLMs work under the hood.

I also know that we data and tech folks will probably never win the battle over anthropomorphization.

The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white lie to get the user to act so that they get a lot of value), and there's really too much to fight against.

We're just gonna keep on running into this and it'll just be like when you take chemistry and physics and the teachers say, "it's not actually like this but we'll get to how some years down the line- just pretend this is true for the time being."

I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood.

The average human is so easily convinced that humans "know", "think", "lie", "want", "understand", etc.

But really it's all just a probabilistic chain reaction of electrochemical and thermal interactions. There is literally nowhere in the brain's internals for anything like "knowing" or "thinking" or "lying" to happen!

Strange that we have to pretend otherwise