Earlier quoted context omitted.
One thing I've been looking at lately is the various strands of engineering (civil, electrical, mechanical). I've been looking for common threads and general principles that apply to software engineering that otherwise haven't been. A part of what inspired that search is what you discuss: a huge amount of rework. Tools that shift like sand dunes. Many principles and best practice based on "this is what worked for us"…
Maybe the other engineers would love to be able to do more rework, but since they deal with hardware, changing things is a lot more expensive. Why do you think that their ways are superior? Construction projects too run way over budget and sometimes even fail, just like software.
I'm feeling for answers. I'm not saying I know what we need. I am, like I think everyone else is, still in the observation stage looking for hypotheses.
I did find evidence based software engineering. This offered some interesting papers. I recommend this. There isn't much and it's far away from a cohesive theory but hopefully we can get there, as a field, soon.
If someone is further along, or has some ideas, I'd love to speak with them.