Ask HN: Is Accidental Complexity Growing Faster Than Essential Complexity?

Sure, for a given project, you can make it more or less complex by taking different approaches. But there's some lower bound on essential complexity and when the product itself requires a lot of features that interact in complex ways, your essential complexity is going to be high. And once the complexity gets too high, every new feature you add starts making future features even harder & slower to add. (A few caveats: 1. You can of course argue that it doesn't really "need" all those features, and everyone would be better off by sacrificing a little on the user experience to make it a lot less complex, less buggy, cheaper to build/maintain, etc. 2. By the time you run into these issues, the program is large enough that some accidental complexity has snuck in, and it can be hard to estimate how much of your problem is from that vs how much is from the essential complexity.)

Anyway, If a project never gets to that size where the essential complexity becomes unmanageable, then great. I'd agree it's probably a mistake to introduce extra complexity with microservices or anything else, and certainly some people make this mistake (or miscalculate whether their project will stay small and simple).

But if it does get to that point, some of these solutions you mention like microservices can be an independent, constant complexity cost. Ideally each microservice itself is a small app with manageable complexity. On the other hand, a naive, monolithic app will continue to have its essential complexity grow super-linearly with variables like time, lines of code, number of developers, etc. The bet is that by introducing the new thing, you can bring the overall complexity of the entire system down.

So my short answer would be "no" - in many large projects, essential complexity is growing faster than accidental complexity. This has led to a proliferation of tools to help bring it under control, and yes, if you look at these tools in isolation they are pretty complex.

(And there are hundreds of billions of dollars worth of companies using these tools, if it is all just a waste of time then there's a massive business opportunity waiting for someone to undercut them)

---

After reflecting on this question, I feel like a good way to think about it might be this:

Companies are hitting extreme points right now. On the one hand you have companies that have stuck to their way of working for years. It worked, and in the process of getting customers and making the money come in, they never really thought to upgrade the company processes. After all, back in the day, tech moved at a much slower pace and that's what they are used to. There's been little to no incremental improvement in the technology or even the practices that have been used throughout the companies life. These types of companies still use FoxPro for ERP's, much much older Java enterprise tech stacks (and practices) for their server side stuff. I don't say that that's bad mind you. The ERP company has been around for over 25 years.

On the other hand you have the fresh companies started by people who really don't want to enter the juggernauts of the market. The establishment. Guided by the excitment that reaches us from Facebook listicles, and Techcrunch, they want to ride their own paths and build a fresh future. They want to go in and show businesses that the establishment is giving them "boring" and that they can do it better. I've met with lots of these people as well. Generally they'll start explaining their business by start from their tech stack. "We are a SME ERP business using react and a full JS based stack". That is not an exaggeration. That is near verbatim.

In between these companies you have the graduates who need to pick a side to go work in. And those that are entering the establishment, want to make their mark. Their impact. They walk into a company that is using SVN for their source code management, and they groan. They see Java being used and they say "why not NodeJS?". And what I've seen that happens, is that they run into the people of the establishment who have no interest in upgrading. Instead of having mentors who work with them towards incremental improvements where each improvement is justified by developer productivity and improvements to the customer they hear "what we have is good enough. Forget it". Or the more common "too much work. Don't bother". As a personal note, the latter really bugs me. Of course it's too much work. You've set it up to be too much work. But, what happens then is that some people will become the establishment, while the others will bide their time waiting for a greenfield project to come and for them to be given a PM role (mind you I see PM roles being handed out to people with almost no technical knowledge, little ability to evaluate tech or specs, after being in a company for 1.5 years). And when they get it, "LET'S DO NODE JS!!!!".

Oh and what of those graduates who left college to join the "entrepreneurs"?. They too have gone through university being taught web app development using "ASP MVC" and "Microsoft SQL" and they long to be let out into the world to play with the tech that they hear their peers are working with around the world. Admittedly, the Open source world of react and angular and what not is super exciting in terms of pace of announcements these days. And then they join their peers and everyone gets to feel excited that they are working with new things because they believe new must mean better.

---

Ultimately the answer is yes. Accidental complexity is growing faster than essential complexity. Business practices change much slower than tech in today's world. I also feel like understanding the reasons behind it is worth pondering on.

For me, I feel that a lack of mentorship has a lot to do with things. There are far too few Uncle Bob's - veterans of the software industry who've kept pace with the change and understand things with a deep historical context - who lead architectural decisions and project management at a company. It doesn't even have to be Uncle Bob level veterans. But my observation of companies from where I am shows me very clearly that after 3-5 years of being in the industry, the deeper level constant learning vanishes. Which means that the older employees and the relatively new employees have stopped growing and the mentorship they provide is based on a tiny amount of work that they've done at some point in their life.

So to recap, we have people who've come out into the industry, fascinated by the flashy stuff (which is fine! That kind of youthful enthusiasm is also needed in the eco system). These people quickly make their way up to the management level within a couple of years, and then get to push the flashy stuff, and it doesn't go beyond there. Understanding the balance between tech decisions and what's required to solve the problem efficiently for the customer doesn't happen. And then these people guide the next generation which multiplies the effect.

From my POV and from where I live, this is the service industry today. Accidental complexity is a growing beast heading towards an exponential curve.

This is true for requirements but also for used technology.