Earlier quoted context omitted.
The "compiler will solve everything" theory seemed to last quite a long time, I first heard it with the Itanium and again with these pipelined CPUs. Narrator: It didn't solve anything.
This is speculation, but: optimizing compilers are pretty good, right? On x86 at least. Perhaps they do a good job on popular platforms like x86, because we can encode decades of experience, but not so great on brand new ones.
I'd call the state of the field quite bad. For example they do embarrassingly little for you to help with the 2 main bottlenecks we've had for a long time: concurrency and data layout optimization. And for even naive model (1 cpu / free memory) there is just so much potentially automateable manual toil in doing semantics based transformations in perf work that it's not even funny.
A large part is using languages that don't support these kinds of optimizations. It's not "C compiler improvements hit a wall", it continues "and we didn't develop & migrate to languages whose semantics allow optimizations". (There's a little of this in the GPU world, but the proprietary infighting there has produced a dev experience and app platform so bad that very few apps outside games venture there)
There's a whole alternative path of processor history not taken in the case that VLIW had panned out, and instead of failing because of optimism about compiler optimizations.