The Wetware Crisis: The Dead Sea Effect (2008)

I grew up working construction on the side with my father, became an appprentice lineman, moved to aftermarket electrical installations in CNC machines, then ended up writing software products for over a decade. I've worked on a wide variety of software: Embedded projects requiring 2000 line assembly ISRs for legacy RS232 protocols, reverse engineering binary network protocols, implementing network server protocols from specifications, writing my own actor and dependency injection frameworks in managed languages, web application backend work, frontend work with JavaScript/TypeScript and frameworks, and so on. I also do a lot of product management, support, customer management, and internal management. Much of my customer support involves teaching external developers from large corporations how to use HTTP and SDKs.

I'm usually one of the smarter people in the room when doing software development and spend a lot of time helping others. In construction, I'm the enthusiastic idiot as the skillset is different. Most tech workers underestimate how high skilled tradework and many other positions outside their experience are, especially management work.

Having said all that, I feel uniquely qualified to offer a rebuttal. The article is describing a clear pattern that I see at large companies that are not focused on software development or lack solid engineering leadership. Ironically, the problem gets worse the more a company tries to outsource IT as they face all the same challenges with less control. It is an organizational health issue however, not some universal truth. If tech workers lack anything, it's experience with organizational growth, change, and group dynamics, not construction experience. Organizations are like people, they grow, change, figure out who they are, have identity crises and need different things in different phases. Ask the highest level/most competent manager you have access to for book recommendations if you're interested in this.

Anyway, back on topic, software "gruntwork" typically implies a department lacks the agency to automate away said gruntwork or lacks the skills to do so. As an example, I work with many organizations using the JVM, but none of them use Scala, Kotlin, Clojure, or any other "nice" JVM language. They use Java. In some cases Java 8. If you're writing Java, there is absolutely stupid gruntwork. I've written example applications for some of these organizations and I had to create code generation tools to stay sane in this environment. In software, you eliminate stupid gruntwork with tools, not people.

We do need average enthusiasts in software development, but it's not the same as construction. In construction the less talented person fetches things and does setup work. In software development, the less talented workers spend most of their time using libraries, plugins, frameworks, compilers, interpreters, databases, and languages while the more talented workers write them.

My experience isn't universal, and I'd be interested in hearing some dissenting opinions on this.

I have spent all week exchanging emails with the "lame duck" in my ISP's accounts department. The task involved should be menial, just copy the bank account name from my email into a form, they are incapable of getting this right.

> Run to the truck and get nails also requires explanation/experience to get right if you have never done it before

After retiring from IT I volunteered at a raptor conservation centre. My previous experience counted for nothing and I was given numerous menial tasks: clean that water bowl, chop down that overhanging branch, sweep out that pond, re-attach that broken perch, put a trailer on the ATV and take this rubbish to the dump area, clean that aviary back wall, operate the music for the 11.30 display, etc.

After doing this for a while I had great experience of how the place actually ran, where things were kept, when it was safe to do things without interrupting flying, etc. I knew I was getting somewhere when my tasks started being looking after work-experience students, inducting new volunteers, helping in the displays, etc. It was actually quite humbling being a complete newbie again and having to go up new learning curves. I wish I had done it sooner in terms of resetting my objectives of what work actually involved.

I’m talking about in the technology function building systems.

100% of the most catastrophic IT disasters I've witnessed have been directly attributable to the most intelligent and skilled programmers I've worked with.

Example 1: I got hired to work in the dev department at a large regional printing company. One of the suites of software built in-house took large retail clients orders for printed store display material. From the client's initial order the software would automatically detail print batch sizes, assign these to the work queue, and would then go on to figure out how many of each type of display needed to be boxed and shipped to each of the client's retail locations and would create shipping labels, packing lists, etc. from this. At it's heart was a 10 page long SQL query that I couldn't make heads or tails of after two weeks spent studying it. I left the company certain of two things: 1. the dev that wrote that was one of the 3 smartest people I've ever met and 2. when he left the company they had literally zero chance of ever hiring someone to maintain that system.

Example 2: I got hired by the 2nd largest print media conglomerate in North America to help manage a fleet of 31 newspaper websites and a fleet of 125 niche verticals. This was at a time when the industry was just beginning to come to terms with their revenue models being gutted by craigslist and the shift to online marketing. To say the least money was tight. It was decided that the back-end systems that drove the integration between newsroom terminals, printroom layout qeues, and the newspapers' websites needed updating.

They hired #2 on my list of top 3 most intelligent people I've ever met to drive a complete overhaul of the system. It was decided a complete rewrite in a cutting-edge language was called for. The guy in charge didn't sleep for 3 days after which he'd provably onboarded and synthesized a complete understanding of the chosen language and it's ecosystem despite no prior experience, then went on to single-handedly rearchitect (feature complete) a system with roughly the same LoC as an operating system. In six weeks.

Over the course of the next 18 months the dev department responsible for implementing this masterwork (no sarcasm) floundered and the project was eventually scrapped having blown through the entirety of it's original 2M budget with literally nothing of use or note to show for it. Apparently nobody else on the team was capable of implementing and debugging the blizzard of microservices the new architecture called for.

My key takeaways: proposals to make codebases "interesting" should be met with deep skepticism if not outright hostility, and a roomfull of mid-level developers are significantly less likely to get an organization into deep trouble than a single rockstar with the bit between their teeth.