> problems that require knowledge that simply doesn't exist yet.
Obtaining the knowledge is a process of trial and error, bruteforce, observation, search, etc. Humans follow that same process. Machines can do that too. If they discover or are taught the same heuristics humans use, and a computational capacity greater than that of all humans combined, they will outpace us in this endeavour.
> Until "AI" turns into general purpose robots that can develop new tools to explore the world, it is, in fact, pretty damned limited in what it can do without human help.
For sure, but this isn't a question of if, but when. LLMs are already rapidly accelerating the pace of every scientific field, so advancements in technology will start to compound.
> Biology is replete with examples. Computers "solve" protein folding [1], and midwits immediately leap to conclusions that drug development will also quickly fall. But we literally have no idea how most of biology works, and simply getting to the starting line for drug development problems is often 95% of the battle. Come talk to me when you've done a million experiments to find the fundamental knowledge that unlocks the pathway(s) we didn't know about that makes a drug discovery program possible in the first place [2].
I don't disagree with you. The amount of complexity in biology is incredible and due to its unpredictable squishy and noisy nature and the advanced machinery needed to analyze it with precision at the microscopic scale, vast amounts of data accumulation are still required to discover the useful patterns here, and the search space will be enormous. Still, it will be done. Here is some recent movement in that direction: https://www.anthropic.com/news/model-hardware-standard-resea...
> I am not pessimistic about humans running out of challenges. We'll just declare one class of problems "done" [3], and move on to the next frontier, as we always have.
This is fair, you're probably right about that. I think they'll be challenges for the AI to solve though, not humans as a collective. Hopefully we have something to contribute to that process, other than just expressing our desires.
> The problem with AI doomers is that they lack imagination that extends beyond computers, or perhaps more accurately, are so sophomoric in their thinking that they skip over the hard parts of any problem they don't fully understand. This stuff reminds me of the endless smartypants whinging about the end of human intelligence when chess machines started beating grandmasters. Chess was never really that great a measurement of human intellectual capacity, and we found new things to do with our big monkey brains.
I'm not an AI doomer, I despise the AI doomer movement and the incessant fearmongering, it's incredibly unhealthy and its only purpose is to perpetuate panic in the aim of maintaining and establishing power, and preventing others from obtaining power that threatens the status quo. What cultish idealogical humans will do with this technology if they're left unchecked with staggering power and no one to stop them is certainly a big concern though.