Earlier quoted context omitted.
Instead of going into a long technical argument of why your description of LLMs is flawed, I'll go straight to the point, because people keep moving the goal posts. What exact problem would need to be solved by LLMs to convince you that they DO discover novel solutions?
unify general relativity with quantum mechanics. The continuum hypothesis. The traveling salesman problem in polynomial time.
"Neural networks will never be able to understand this sentence that's obvious to humans"
to
"LLMs must be able to solve problems that humanity hasn't been able to after almost a century, and that might even be unsolvable"