I've had the same ideas, that the situation with AI or any scientific tools in the gaming mainstream is unsatisfactory. I had to deal a lot with AI air traffic control in MSFS2004 and it was noteable that they abandoned it's development as soon as it was able to do all the chain of basic operations from taxiing and takeoff to approach and vacating the runway. The real-life ATC are always sloving an optimisation problem, which is the most crucial at approach. Of course, AI ATC doesn't care to optimize anything and may make a poor a/c go around forever.
One problem with AI is that it can't be estimated quickly by game reviewers. It takes much time to notice it's patterns, probably it takes to complete all the missions up to the end. As a result, AI quality doesn't affect game's grades much enough to care about it.
Another thing is that many game developers are good at algorythms and programming, but not (if at all) at game theory, and social math theories (like evolutionary dynamics), and optimal control.