A large part of the problem is that grand majority of software (at least if it's facing actual human clients) is constantly trying to search or validate market-fit. If you don't get it out first, you're going to lose to a competitor who does.
And even when you do have a solid market-fit figured out, you still don't know what your customers will actually want. (And don't get me started on customer research. People lie all the time, and their accuracy on what features are honest-to-$deity blockers.. Nope.) This means that for most parts the incentives are misaligned and work actively against delivering high-quality software.
Underlying the agile manifesto is the need for engineering to be properly aligned with business incentives. Continuous delivery, fast iteration cycles, incremental improvements and rapid deployments are all facets of the same fact: you don't actually know what your customers want (because your customers don't know what they want!), and the only true way to figure it out is to iterate as fast as you can.
It's only when you have an established moat and a practically guaranteed, net-profitable income stream, when you can even imagine at having the opportunity of doing things the way you would assume you want to.
Oddly enough.. the companies who have such moats and guaranteed profits tend to be the same ones that have been the slowest to modernise themselves. Banks - until the modern online-only challengers came along. Insurance, where the regulatory moats are even higher and is only now being "disrupted". Loan brokering. Energy. Water. Public transport (where monopolies and/or charters prevent competition). Healthcare. On and on and on...
And even then, every one of these established, guaranteed-profit machines has the same problems: the moment one good new player enters the field and starts stealing customers, the old guard needs to find a way to either adapt - which is the best-case scenario - or rely on their power structures with politicians and quickly require new regulations to strangle the usurpers.
Software is expensive to make. It's only cheap to deliver. And time spent making the wrong thing is really expensive.
EDIT: I forgot to say this the first time around. Engineering career incentives are also misaligned. Maintenance is not appreciated, only shipping new stuff is. So the developers who would want to improve what exists and make it better will be looked down upon. The promotions and raises go to those who ship something new, even if it ends up setting the world on fire a year later.