You definitely should not make claims like 'LLMs can't drive cars'. LLMs have already been shown several times to be able to navigate as agents in different world environments. Obviously, I'm not advocating this is a good idea, and certainly light years away from safe - but as an experiment in silico, I imagine it can be done quite easily (and probably already has).
The reason scientific researchers in this area are using the term 'AGI' so much is that it does fit the definition of AGI ... *For some definition of AGI*. And there lies the problem - no one can really come to a good consensus on a good definition of AGI. This is why many scientists in this area are avoiding the question altogether - the question is loaded, and is misinterpreted by the public if statements are made.
So, for example, if I make the statement here that e.g. GPT-4 has intelligence which is general (AGI), it will likely be met with a rabid response from HN. However, the claim may be more dull than you're expecting. People often conflate AGI with things that are not required, such as agency, etc.
This definition from [journal Intelligence Vol 24, No1, 1997] can be that "some definition" of AGI: must be able to 1) think abstractly, 2) comprehend complex ideas, 3) reason, 4) plan, 5) solve problems, 6) learn quickly from experience.
Many of these GPT-4 can do, if some modifiers are allowed - for example, GPT-4 can learn quickly from experience so long as you aren't starting a 'new' GPT-4 system from scratch every time you want to interact with it. This is probably preferable, since much of the experiences it will have are personal to the individual working with it, and it would be highly undesirable to do the opposite here.
Planning was difficult for the system early on, but it has appeared to learn that it is helpful to lay out a plan early on in a large task, so that appears to be a capability as well, at least on a basic level qualitatively.
There is some good literature on ability to reason (and solve problems in abstract and complex ideas) from Microsoft's group on causal reasoning. In pretty much all tasks of causal reasoning the system can achieve near human performance, and LLMs as a category outperform previous SoTA from more targeted or specific systems made for causal reasoning.
Anyway, I would suggest to anyone that has strong reactions to claims about AGI to realize that they are likely building up the statement to be more than it is. Perhaps similar to 'machine learning' may have been misinterpreted years ago ("A machine can learn?! There is no tomorrow!!"), what is being stated here is often more narrow than you may believe.