Great article! I've been thinking about this problem since 2006, and there are probably a hundred reasons why things are as bad as they are. Many commenters here have hit upon some: misuse of abstractions (they should be used to save time, NOT save knowledge), insistence on code reuse, etc. To add a few more: treating programming like a form of manufacturing instead of an art, believing that the latest language/tool/process is a silver bullet (and even believing that it's new!), not realizing/accepting that constant refactoring is vital, worshiping teams at the cost of individuals, not letting teams form themselves, and a huge one is this: trying to
eliminate problems and/or complexity instead of
mitigating them with tools and skill-building. If you try to
eliminate a problem or complexity, you will succeed, but at the cost of creating at least as many other
different problems than you had before, and you'll just end up moving the complexity to another place. Later, when you realize you still have problems, you'll try the eliminate them too, and do the same thing all over again. Building layer after layer, always increasing your problems and complexity while attempting to reduce them. (This is exactly the same as government controls breading more controls, and it happens for the same reasons.)
When I first started out, if you were programming computers at all (microcomputers, anyway) you were someone who was naturally curious and probably had a high IQ. If you continued programming, it's because you really enjoyed the process and had a knack for it. You understood how the computer really worked, and that didn't scare you, it thrilled you. We didn't have "testers" and "UX Designers" and "QA people" back then, so if you produced a good program at all, it's because you had a high level of conscientiousness. Today, that description fits only a tiny minority of the programmers that I know, just like it fits only a tiny minority of the general population. Those programmers all have the same complaints that you do, and they all have side projects that they work on because the work in their day jobs is unfulfilling.
As someone here pointed out, this is a relatively new industry, and I think we're just seeing the same thing that plays out in any industry once "everybody" starts doing it. Just like vacuum cleaners, mattresses, doctors, or oil paintings, there are a handful of great ones, a whole bunch of mediocre ones, and some really awful ones. The bell curve has arrived and is here to stay!
My belief is that if you want to have great software, then make it yourself. Don't wait for the majority of others to produce it, because they never will.