Another reason I'd like to add regarding why so many software projects fail is a really basic one, but is ultimately the reason why most software projects fail.
The budget isn't there.
For many, building software is a race to the bottom. I've worked at countless places, from "Wagile" places that run spiral methodologies mixed with agile, to fully agile agencies that deliver well but crumble the second a client gets pissy about something taking longer/costing more than it should.
In my view, the most basic problem in software is that we're committing to too much for too little, which is why I see development to be similar to working in a skilled trade. If you pay good money for a renderer, you'll get the outside of your house rendered nicely with good advice on what to use, what looks good. They'll also tell you how long it'll take, and if you say you want it sooner they'll tell you it'll either cost a lot more to get more manpower, or they'll decline the job. If you are cheap about it, you'll probably get someone that'll take longer than expected, will make a mess of the job, and you'll be left with something you're not entirely happy with.
A solid methodology will probably help with delivering software on time and on budget, but if you are unrealistic with either metric then it doesn't matter what methodology you use. You'll take liberties with it, decide that it's bullshit, and continue to cowboy your way towards a duct-taped mess of a solution.
It's something few want to talk about, probably because there isn't really a solution to it outside of:
* Paying a premium for a development team with a track record of recent success
* Having people that know the full software lifecycle be involved in all parts of the process
* Actually embracing the fact that when requirements change, to the point where budgets and timescales are flexible.
* Not joining the race to the bottom.