People are just as bad as my LLMs

You are correct in that the date thing by itself, if that was the only thing would not be such a big deal.

But the date thing and confidently telling me the wrong date is a symptom and stand-in example of what LLMs will do in way too many situations and regular people don't understand this. Like I said, not very intelligent / confident people will do the same thing. But with people you generally have a "BS meter" and trust level. If you ask a random stranger on the street what time it is and they confidently tell you that it's exactly 11:20:32 a.m. without looking at their watch/phone, you know it's 99.99% BS. (again, just a stand in example, replace with 'Give me timeline of the most important thing that happened during WWII on a day by day basis' or whatever you can come up with). Yet people trust the output of LLMs with answers to questions where the user has no real way to know where on the BS meter this ranks. And they just believe them.

Happened to me today at work. LLM very confidently made up large swaths of data because it "figured out" that the test env we had was using the Star Trek universe characters and objects for test data. Had no base in reality and it basically had to ignore almost all the data that we actually returned from one of these "Get the current date" type functions we make available to it.

Thanks LLM!

On a trivial level, it's obviously true that every token in an LLM's output must have existed in the training data. But that's as far as your observation goes.

The point is that LLMs can produce novel sequences of tokens that exhibit the functional equivalent of "understanding" of the input and the expected output. Further, however they achieve it, functionally their output compares well to output that has been produced by a reasoning process.

None of this would be expected if they simply repeated "a bunch of training data predictions averaged together... or something roughly like that." For example, if that were all that was happening, you couldn't reasonably expect them to respond to a prompt with decent, working new code that's fit for purpose. They would produce code that looks plausible, but that doesn't compile, or run, or do what was intended.

One reason your model of the process fails to capture what's happening is because it's not taking into account the effects of latent space embeddings, and the resulting relationships between token representations. This is a major part of what enables an LLM to generalize and produce "correct" output, taking meaning into account, beyond simply repeating its training data.

As for intelligence - again the question comes down to functional equivalence. If we use traditional ways of measuring intelligence, like IQ tests, then LLMs beat the average human. Of course, that's not as significant as might naively be imagined, but it hints at the problem: how do you define intelligence, and on what basis are you claiming an LLM doesn't have it? Ultimately, it's a question of definitions, and I suspect it'd actually be quite difficult to give a rigorous (non-handwavy) definition of intelligence that an LLM can't satisfy. This may partly be an indictment of our understanding of intelligence.

In my opinion there's nothing wrong with the traditional definition which is "the ability to acquire and apply knowledge and skills". But if you want to reach a minimum of "hand-waviness" then it's additionally required to define 'acquire', 'apply', and 'skills'. My personal definition is that acquiring knowledge requires building some sort of internal semantic model of it, though the occurrence of which there is actually evidence of in LLMs (see "abliteration"), so one out of three so far. But it falls apart at 'apply'. How do we even define applying? Well, I do not define it as what LLMs do, which is to predict the next token of the data.

I, personally, apply my knowledge by recognizing where it may be applicable, bringing it to thought, and then using that in the construction of ideas or strategies that I can act on. There's a degree of separation here between thought and action that doesn't currently seem to exist in LLMs; some creators are trying to simulate it by having an LLM for thoughts and another LLM for actions, or by enabling the thoughts to call tools that perform actions, or by having the LLM think before acting as in DeepSeek R1, but that isn't quite it.

An LLM still doesn't understand, say, spatial reasoning when it is helping me write something like a battle in a story. I have spatial reasoning because I can literally see what is happening while I write. I can see, and feel, and hear, and everything. Maybe that's just my dissociative disorder, but I will continue to await the day where LLMs might be able to do stuff like that. Until they can have that essentially happening "in their head", reason about it, and write using that, I won't believe that LLMs can "apply" much of anything just yet. (Other than machine learning I guess.)

> If we use traditional ways of measuring intelligence, like IQ tests, then LLMs beat the average human.

I think the whole notion of IQ is flawed because of neurodivergence. To put things in vaguely ableist-sounding terms (I don't mean it that way, but it will always sound that way), LLMs right now feel too neurotypical (pattern-based) and I would like to see future LLM developments that allow models leaning closer to autistic (logic-based).