A friend of mine nailed the problem. His theory: for most apps the user is the commodity; not the software!
This is the number one reason why usability of software is not improving and often even declining. The other is feature creep. I'll get to that later.
Caveat: I will be generalizing a lot. :)
For the first point —
In the case of commercial software this is obvious. You get paid an hourly rate and the company buys software that you then use. Aka: you are not the one deciding which software to buy. How much you enjoy the experience of using it and how productive your are while doing so is thus not really important for the vendor.
In the case of open source software you commonly do not pay for the software too. The software is developed by people for various reasons. They may use it themselves or they just like working on it. Again – if you have an issue they do not really have reasons to care that much.
Feature creep —
One of the things that makes a developer very happy is adding a feature to a software and exposing it to the user.
As a developer myself I know the feeling. It's wholesome, warm, fuzzy.
But when you expose a feature you need to add a user interface to this feature. This is the most difficult part.
The number of parameters driving a feature is also called a 'parameter vector'. The more publicly exposed dimensions such a vector has the more difficult the feature is to use.
A feature that has ten parameters may be useful to 99% of users if only three of these ten are exposed. The rest can have magic numbers in the code.
Adding another seven dimensions to the publicly exposed parameter vector of a feature to cover 100% of use cases is a bad idea.
Deciding these things requires intricate understanding of the problem space from the user's side. Most developers are not good at this.
And more parameters added to a UI somehow feels better to most people. Even though they do understand that this can be counterproductive.
So my friend had this analogy: imagine if you got paid to use a mobile phone. Imagine your government bought phones from Apple, Samsung; whoever. And then you got paid for using them.
Do you think we would have something like an iPhone or modern smartphones? Unlikely. We would have crazy awful phones from the pre-smartphone era. Probably with much worse UX than some Nokia or Ericsson phones had at the time just before the iPhone appeared.
This is the situation we have with most of the common closed- and lots of open source software. Again: I am generalizing here.
On the bright side: software that needs to fight for its user base – whether open or close source –, often has better usability.
And certainly: for everything I said above there are countless counter examples. But the overall trend seems obvious to me.
I agree with the author of the article 100%.
Well, maybe I'm just grumpy and old too. :]