Artificial intelligence systems found to excel at imitation, but not innovation

Arguably the human-ness (or even animal-ness) of thought is, at its root, characterized as the ability abstract in a novel manner (the neurocog term for innovate).

In other words, it is the ability to find patterns between two concepts that weren't, ever, prior announced. A computer that could do this would then be AI. A computer that cannot would fall short of that category, however otherwise dazzling in stitching together established patterns.

A quick human, and therefore, machine test of this ability might be decoding of prior unseen allegory. How fast and accurately can a human or machine identify (abstract) any true pattern in an allegory that can be applied to another seemingly unrelated concept or story? Again, this would have to be a new allegory to the subject. Ideally, harboring a pattern that isn't discussed anywhere in training data.

The AI would be held to be improved as its ability improved to decode and apply increasingly abstract or otherwise complex patterns from allegory or stories.

Not that I would be, but I'm un aware of any LLM / AI progress toward that type of processing.

As a human, I often find innovation harder than imitation as well!

* I do love trying to innovate, and it feels so good when it's a success!

Hopefully.

However, I had the impression there doesn't exist enough training data to make that place different in a meaningful way.

Still, I think, letting some skilled UX designers loose on input methods could improve things quite a bit, even if the models won't get "smarter".

Known feedback is already encoded in the inputs. I don't remember move 37, but I think I know to what you're referring. Playing games against itself, and remembering, meant that outcomes (i.e. feedback) were already encoded in the inputs for the next run. It had a goal - win the game - but it's hard to claim that the machine itself was wilful. The people who designed it were. But, the machine was just an incredibly effective goal-seeking instrument. The rules were clear, the goal was clear; everything was pretty constrained. Just because move 37 never happened to have been used previously in the recorded history of Go, except in the games the machine played against itself, is meaningless in a claim to innovation. It was just both novel and effective. The will sets the goal, it precedes it. The scientific method is neither here nor there. I don't understand its relevance to the point I believe you're trying to make. Theorize, test, refine or abandon. It's mechanical, except in the theorization and the decision making around refinement or abandonment. What can we imagine? and how shall we judge? Both efforts of will. Someone has to decide that shelter is a good thing, and some smart person in Zambia, a long time ago, came up with the idea and pushed it forward. That person must have wanted to do it, without prompting and without prior knowledge; at least, the first one must have. I recall Einstein's own story that he came up with the original theory of relativity by imagining what it would be like to sit on a light beam. Innovative stuff. Imagining doesn't seem very AI-ish to me. My personal view is that, right now, AI's have only memory, and that they don't have experience. Experience is the why?-part and the judgement part without an external entity establishing a goal-seeking mechanism. Maybe they'll get there. Who knows?

Why would a system (LLMs in this case) designed to produce the most likely next output be innovative?

A truly innovative AI would probably hallucinate or produce so many nonsensical outputs as to be deemed useless and broken.