LLMs are an incredible technological breakthrough which even their creators are using completely incorrectly - in my view, half because they want to make a lot of money, and half because they are enchanted by the allure of their own achivement and are desperate to generalize it. The ability to generate human-style language, art, and other media dynamically on demand, based on a prompt communicated in natural language, is an astonishing feat. It's immensely useful on its own.
But even its creators, who acknowledge it is not AGI, are trying to use it as if it were. They want to sell you LLMs as "AI" writ large, that is, they want you to use it as your research assistant, your secretary, your lawyer, your doctor, and so on and so forth. LLMs on their own simply cannot do those tasks. They is great for other uses: troubleshooting, assisting with creativity and ideation, prototyping concepts of the same, and correlating lots of information, so long as a human then verifies the results.
LLMs right now are flour, sugar, and salt, mixed in a bowl and sold as a cake. Because they have no reasoning capability, only rote generation via prediction, LLMs cannot process contextual information the way required for them to be trustworthy or reliable for the tasks people are trying to use them for. No amount of creative prompting can resolve this totally. (I'll note that I just read the recent Anthropic paper, which uses terms like "AI biology" and "concept" to imply that the AI has reasoning capacity - but I think these are misused terms. An LLM's "concept" of something bears no referent to the real world, only a set of weights to other related concepts.)
What LLMs need is some sort of intelligent data store, tuned for their intended purpose, that can generate programmatic answers for the LLMs to decipher and present. Even then, their tendency to hallucinate makes things tough - they might imagine the user requested something they didn't, for instance. I don't have a clear solution to this problem. I suspect whoever does will have solved a much bigger, more complex than the already massive one that LLMs have solved, and if they are able to do so, will have brought us much much closer to AGI.
I am tired of seeing every company under the sun claim otherwise to make a buck.