> Why do people say stuff like this that is so demonstrably untrue? SD and GPT4 do not exhibit the behavior described above and they're not even new.
it's true that most people do not actually understand the problem/limitation, but it's a discussion that is statistically likely to occur on the internet and therefore people tend to regurgitate the words without understanding the concept.
I'm being facetious but honestly it's a major theme of this whole AI revolution, people do not want to accept that humans are just another kind of machine and that their own cognition resembles AI/ML in virtually every aspect. People confabulate. People overreach the bounds of their expertise. People repeat words and concepts without properly understanding the larger context in which they need to be applied. Etc etc.
Has nobody ever watched someone get asked a big question or an unexpected question and "watched the wheels turn", or watched them stammer out some slop of incoherent words while they're processing? Does nobody have "canned responses" that summarize a topic that you can give pretty much the same (but not exactly, of course) every time you are asked it? Is that not "stochastic word chains"?
By design neural nets work almost exactly the same as your brain. But a lot of people are trapped in the idea that there must be some kind of "soul" or something that makes human cognition fundamentally different. By design, it's not. And we don't fully understand the exact modalities to encode information in it usefully and process it yet, but that's what the whole process here is about.
(I commented about this maybe 6 months ago, but the real hot take is that what we think of as "consciousness" isn't a real thing, or even an "overseer" within the mind - "consciousness" may be exactly the thing people mean when they say that "LLMs have to write a word every time they think about a concept". "Consciousness" may in fact be a low-dimensional projection of the actual computation occurring in the brain itself, rationalizing and explicating the symbolic computations of the brain in some form that can be written down and communicated to other humans. "Language" and "consciousness" as top-level concepts may actually only be an annex that our brain has built for storing and communicating those symbolic computations, rather than a primary driver of the computations itself. It's not in control, it's only explaining decisions that we already have made... we see the shadows on the wall of plato's cave and think that's the entire world, but it's really only a low-dimensional projection.)
(or in other words - everyone assumes consciousness is the OS, or at least the application. But consciousness may actually be the json serializer/deserializer - ie not actually the thing in control at all. Our entire lives and decisionmaking processes may in fact be simple rationalizations and explanations around "what the subconscious mind thinks should happen next".)