I resonate with much of what you have pointed out, but I have a different perspective on certain parts.
Software consists of various layers, and everyone has their own specific areas of strength.
For instance, because I am an application programmer, I often need to write code that prevents the program from halting—aligning with recent programming trends that involve preserving the computation context using monads.
In other words, my strengths lie in the overall architecture and interface design.
This fundamentally relates to cohesion and coupling. I excel in this area, particularly when dealing with codebases around the 60,000-line mark. This is a realm where books like Clean Code are quite effective (many people dislike it, but it is actually a well-written book).
Put differently, I possess the ability to mass-produce software (regardless of absolute quality). Over the course of 7 years, I have built CRUD applications for 43 companies across 16 different domains (ranging from drones and golf simulators to tax SaaS and supermarket POS systems). Therefore, I believe I have at least an average, solid capability in this regard. The primary area where I actually made money was PLC, so while I may not have deep academic expertise, I certainly do not think I lack capability.
In Korea, the profession known as SI (System Integration) is a field where you enter contracts on a "project" basis. In that environment, I have encountered a wide variety of people. From those experiences, my takeaway is that programming is divided into quite several distinct layers.
To speak of algorithms first: I learned basic algorithms and fundamental data structures in university. However, in the field where I worked, there were many people who struggled to implement those basic algorithms, yet they still built a large number of applications. Why is that? There was even someone who made an amount of money I could never dream of touching in my lifetime. Why did he make so much money when he didn't even know how to implement basic algorithms?
The answer is quite simple. It is because the "implementation model" and the "contract and cost model" are different.
The contract and cost model is a self-contained body of knowledge. It is the ability to know exactly where to fit a given piece into the puzzle. Implementation is simply the ability to build that piece from scratch. In fact, mostly due to issues like employee turnover and the organization's future maintenance capabilities, many teams (specifically, organizations with lower implementation capabilities) decide on an open-source library and design their architecture based on its API. In these cases, the primary technical challenge becomes how to connect those components based on the performance of that library.
Yet, people tend to think that only those who can implement from scratch are capable of programming. A person who can take someone else's implementation and piece it together to fulfill their own contract is also a programmer, but people frequently forget this. Depending on which layer you exist in, certain knowledge requires you to implement it yourself, while other knowledge only requires you to understand the contract. I believe this is the core of programming.
Those on the side that must design and build libraries or frameworks naturally have things they must know about implementation, as they are creating the SDKs. However, I have seen quite a few cases where these very people have no idea how their work is actually utilized in the upper layers.
And these types of knowledge are highly fragmented.
In my case, I am familiar with quite a few paradigms. On my personal homepage wiki, I can differentiate between OOP, DOD (Data-Oriented Design), and others, and in the context of relational databases, I know exactly where the ORM impedance mismatch occurs. However, this is largely an area intertwined with architecture, and its essence is closely linked to David Parnas's theory of information hiding.
In other words: to what extent do we hide the internals, and where do we expose them to minimize the contact surface area and ensure a safe connection?
For example, I can't implement PostgreSQL's B-tree. But I can design a business system using PostgreSQL. I only know the name of TCP congestion control—I don't know how to implement it. But I can build networked applications.
That's what an 'industry' really is. The ability to trust the contracts of other people's implementations and assemble them. The ability to trust others.
In that regard, I agree with many of your points. However, I actually believe the software industry needs more of those "average" people. I think the very definition of an "industry" should premise that average people can maintain their livelihoods simply by dedicating themselves to a single specific field. From that perspective, when building one's expertise within this fragmented landscape of knowledge, it is perfectly natural not to know much outside of your own specific layer.
p.s https://www.makonea.com/en-US/casual/cargo-cult-programming-...