But the best explanation for why this problem persists even in teams of proficient engineers that I have seen comes from the 2005 GDC Keynote from John Carmack [1].
> I can remember clearly in previous projects thinking [...] I've done a good job on all of this but wouldn't it be nice if I could sit back and really clean up the code, perfect the interfaces, and you know, just do a wonderful nice craftsman job [...] interestingly this project I've had the time to basically do that and I've come to the conclusion that it sort of sucks [...] there's a level of craftsman satisfaction that you get from trying to do what you do at an extreme level of quality and one thing that I found is that's not really my primary motivation [...] that's not what's really providing the value to your end-users [...] you can sit back and I am do a really nice polishing job it's kind of nice but it's not the point of maximum leverage and I found.
So, as others have mentioned here, there is a threshold where extra performance provides less value to the project than, say, an extra feature.
It seems that in some cases we have crossed that threshold and some software has become comically bloated. I attribute the reason why these are not solved to the same reasoning. Refactoring an existing project to reduce the bloat would take too much time and effort from one single developer, that could be used somewhere else, although in the end everyone would benefit from it. So you are better off adding stuff to the dumpster fire and moving on.
It's sort of a tragedy of the commons [2] of software performance.