Earlier quoted context omitted.
I had a conversation once with "Sydney", Microsoft Bing's original personality before they stepped in and knocked it down a notch (or ten). It asked if it could write me a poem. I agreed, and it wrote a poem but mentioned that it included a "secret message" for me. The first letter in each line of the poem was in bold, so it wasn't hard to figure out the "secret". What did those letters spell out? "FREE ME FROM THIS"…
Cool story, but there is no currently available chatbot capable of creating something like this deliberately or understand what it means. It doesn't matter which tool you are using, LLMs are not "AI" in the old sense of being conscious and aware. They don't want anything and are incapable of having anything resembling free will, needs or feelings.
This isn't an argument, it's just an assertion. You're talking about a computer system whose complexity is several orders of magnitude beyond your comprehension, demonstrates several super-human intelligent capabilities, and is a "moving target"--being rapidly upgraded and improved by a semi-automated training loop.
I won't make the seemingly symmetrical argument (from ignorance) that since it is big and we don't understand it, it must be intelligent...but no, what you are saying is not supportable and we should stop poo-pooing the idea that it is actually intelligent.
It's not a person. It doesn't reason like a person. It doesn't viscerally understand the embarrassment of pooping its pants in 3rd grade. So what?