the comment which spawned this chain is:
> If push came to shove we might just consider making better software?
as you say, "a _lot_ of software out there runs in the 'hardware is my driving cost' locale." if something changed -- say, we lost access to high-end fabs -- then my argument is that we have room to focus higher on the stack to make this far less impactful than it would otherwise be. in most places, software is not "optimal", as you seem to agree. in very few places have we even calculated a bound of what optimal _is_. the HPC example was to highlight the latter part: we shift from something like a linear gradient descent to some different descent algorithm and it works better in practice. ML training techniques are always changing to get faster convergence. very, very few of these HPC areas have actual, well-defined performance bounds. it's just a tradeoff of "how much do we invest in algorithms" v.s. "how much do we invest in hardware". i read GP as suggesting otherwise, based on the content of the comment they were replying to.
i'm not even sure there's a disagreement between you and i. maybe not even between me and sneak. i kinda wonder if all of us here just have a different number in mind for what qualifies as "mostly quite efficient". i'm completely confident we have at least 2x perf gains sitting around in just about any computational area that's presently dominated by hardware costs (e.g. HPC, but not e.g. packet-switching/networking). 10x perf gains in many of them: not by micro-optimizing things like memory allocations, but by being inventive with algorithms. that kind of gain is meaningful to me in this area: enough that i wouldn't dismiss it as "mostly quite efficient" as a rebuttal against that original "if push came to show we might just consider making better software".