What happens when people don't understand how AI works

Sounds a bit like humans. Much data modified by "don't hit your sister" etc.

There's a really fascinating article about this from a couple years ago that interviewed numerous people working on data labeling / RLHF, including a few who had likely worked on ChatGPT (they don't know for sure because they seldom if ever know which company will use the task they are assigned or for what). Hard numbers are hard to come by because of secrecy in the industry, but it's estimated that the number of people involved is already in the millions and will grow.

https://www.theverge.com/features/23764584/ai-artificial-int...

Interestingly, despite the boring and rote nature of this work, it can also become quite complicated as well. The author signed up to do data labeling and was given 43 pages (!) of instructions for an image labeling task with a long list of dos and don'ts. Specialist annotation, e.g. chatbot training by a subject matter expert, is a growing field that apparently pays as much as $50 an hour.

"Put another way, ChatGPT seems so human because it was trained by an AI that was mimicking humans who were rating an AI that was mimicking humans who were pretending to be a better version of an AI that was trained on human writing..."

Some features of animal sentience:

* direct causal contact with the environment, e.g., the light from the pen hits my eye, which induces mental states

* sensory-motor coordination, ie., that the light hits my eye from the pen enables coordination of the movement of the pen with my body

* sensory-motor representations, ie., my sensory motor system is trainable, and trained by historical envirionemntal coordination

* heirachical planning in coordination, ie., these sensory-motor representations are goal-contextualised, so that I can "solve my hunger" in an infinite number of ways (i can achive this goal against an infinite permutation of obstacles)

* counterfactual reality-oriented mental simulation (aka imagination) -- these rich sensory motor representatiosn are reifable in imagination so i can simulate novel permutaitons to the environment, possible shifts to physics, and so on. I can anticipate these infinite number of obsatcles before any have occured, or have ever occured.

* self-modelling feedback loops, ie., that my own process of sensory-motor coordination is an input into that coordination

* abstraction in self-modelling, ie., that i can form cognitive representations of my own goal directed actions as they succeed/fail, and treat them as objects of their own refinement

* abstraction across representation mental faculties into propositional represenations, ie., that when i imagine that "I am writing", the object of my imagination is the very same object as the action "to write" -- so I know that when I recall/imagine/act/reflect/etc. I am operating on the very-same-objects of thought

* facilities of cognition: quantification, causal reasoning, discrete logical reasoning -- etc. which can be applied both at the sensory, motor and abstract conceptual level (ie., i can "count in sensation" a few objects, also with action, also in intellection)

* concept formation: abduction, various various of induction, etc.

* concept composition: recursion, composition in extension of concepts, composition in intension, etc.

One can go on and on here.

Decribe only what happens in a few minutes of the life of a toddler as they play around with some blocks and you have listed, rather trivially, a vast universe of capbilities that an LLM lacks.

To believe an LLM has anything to do with intelligence is to have somewhat quite profoundly mistaken what capabilities are implied by intelligence -- what animals have, some more than others, and a few even more so. To think this has anything to do with linguistic competence is a proudly strange view of the world.

Nature did not produce intelligence in animals in order that they acquire competence in the correct ordering of linguistic tokens. Universities did, to some degree, produce computer science departments for this activity however.

i copy pasted my comment and your question to chatgpt, so this isnt my answer but the AI's:

make your own conclusions

Because both LLMs and the I Ching function as mirrors for human interpretation, where: • The I Ching offers cryptic symbols and phrases—users project meaning onto them. • LLMs generate probabilistic text—users extract significance based on context.

The parallel is:

You don’t get answers, you get patterns—and the meaning emerges from your interaction with the system.

In both cases, the output is: • Context-sensitive • Open-ended • Interpreted more than dictated

It’s a cheeky way of highlighting that users bring the meaning, not the machine (or oracle).

Agreed. Many animals without language show evidence of thinking (e.g. complex problem solving skills and tool use). Language is clearly an enabler of complex thought in humans but not the entire basis of our intelligence, as it is with LLMs.