Striking the Right Balance: Over-Engineering vs. Under-Engineering Software

I notice there are definitely very different styles of thinking (obviously) but it really becomes clear when I read articles like this.

I would say that nothing in this article is interesting or new. IMO, it is not helpful in literally, any single way. (That's not me trying to be rude, I believe this is more-so a function of how people perceive information, and there are certain types of information that click vs not... ie, the wrong kind of fuel).

If I had to pick the right balance on any of the projects or companies I start... it always begins with pacing around my house in a deep sort of trance... pacing around endlessly for hours, standing in the shower while I mumble over and over, trying to visualize the entire "problem space." Of course with complex problems you can't do that, so in a sense the visualization I have will undoubtedly "fade" and I must rebuild it again in my mind, again, and again, and again. Each time trying to iteratively hold more in my mind simultaneously, all the different interwoven layers, without it crumbling.

I also use a huge amount of notebooks to draw different diagrams of the problem, usually messy to start but then becoming more refined as I start to slice and dice the different dimensionality of the problem in different ways. How to hierarchize (is this a word?) the different facets, how to split the functionality up cleanly and elegantly.

For me, anyway, I always focus on just 1 word: Elegant, as I feel like that's a great "sweet spot" for me. I try to find elegant solutions to hard problems. I like this word a lot because there exists solutions which cleanly slice the dimensionality of information, either by jiggling the information in slightly different ways (which has a net positive effect on the end user), and the cost/benefit is such that it improves the business side as well. Always a sort of cost/benefit, cause/effect energy going on. Shifting these building blocks, (which also equates to the actual code/features) until there is a nice fit. A good example might be org structure for a company that has all kinds of reselling/affiliate/white labeling options, and how accounts and financials/books are set, where they live, at what level, etc.

They will undoubtedly become more complicated later, as more layers of features and such get piled on, but I just intuitively piece through those scenarios not by reading articles like this and being like "GEE I GUESS I BETTER INCREASE AGILITY!" but by just feeling out the energy of the problem in my mind, poking at it, following it around like a hard to catch elusive rabbit... slowly hiding behind bushes and trees trying to sneak up on it. This is the idea of chasing the elegant solution which almost always (for me) is very painful and hard to find, with a great amount of stress and energy (unable to sleep or eat until I find the solution). It will haunt every waking second of my mind until I have it, and when I do it hits me like a freight train and I exclaim (!FUCK!) and I quick write it down, and the process repeats, usually from 7-30 days depending on how hard the problem is, how hard the app is, how hard the idea is.

From this way, I've nailed the balance of over/under-engineering nearly perfectly in each of my later projects, a few of which have become absolutely immense

It's interesting to try to fit what is often talked about as "future-proofing" and "reusability" into the development of a general-purpose CPU, since CPUs are in a sense the ultimate reusable system.

In an overly simplified textbook example of designing/building a CPU, you have an ISA you're building the CPU to support. The ISA defines a finite set of operations and their inputs, outputs, and side effects (like storing a value in a particular register). Then you build the CPU to fulfill those criteria.

In my experience, designers that want reusability usually don't have enough precision in how they want to reuse a system so an ISA-like design can't be created.

And practically, it's the rare (I might even say non-existent) day-to-day business problem that needs CPU-like flexibility. Usually a system just needs to support a handful of use cases, like integrating with different payment providers. An interface will suffice.

I very much agree with this. Over-engineering and under-engineering are poorly named as they are not on opposite ends of the spectrum. They are actually both badly-engineered and actually both lead to many of the same issues (which I believe this article gets wrong). Of the listed cons both overlap on all of the following:

- Fragile code

- Technical debt

- Reduced Agility

- Understanding Complexity

Over-engineering can be abused an excuse for poorly-engineered solutions and cutting corners. The future being hard to predict is often used as justification, but this swings both ways, you also often won't know what code is going to get built upon. Frequently an obscure one-off piece of code can become more useful than expected, with functionality tacked on over time, until the point where an entire product is resting on really shaky foundations.

Build a culture of quality engineering. Build the minimal solution but build it well. Have a strong (and flexible) product vision as a guiding light but always take small steps towards it. Optimize towards understandability and replaceability.

This is possibly the best description of my development process I’ve ever seen.

I’d add “staring into the distance” alongside the mumbles (yes, when pacing, that often means I run into corners of walls, couches, etc).

I think of it as iterative development in my head, and it’s often focused on finding the fastest path to proving that an architecture or implementation is wrong; when I find it hard to prove why something won’t work, then I begin to explore in depth the idea and have a pretty big running start on the code, which flows a lot more quickly when I’ve done this than when I just sit down and start typing.

Earlier I wrote about the "Simulator Effect" aka "apophenia", and "Reverse Over Engineering":

https://news.ycombinator.com/item?id=22062590

DonHopkins on Jan 16, 2020 | parent | context | favorite | on: Reverse engineering course

Will Wright defined the "Simulator Effect" as how game players imagine a simulation is vastly more detailed, deep, rich, and complex than it actually is: a magical misunderstanding that you shouldn’t talk them out of. He designs games to run on two computers at once: the electronic one on the player’s desk, running his shallow tame simulation, and the biological one in the player’s head, running their deep wild imagination.

"Reverse Over-Engineering" is a desirable outcome of the Simulator Effect: what game players (and game developers trying to clone the game) do when they use their imagination to extrapolate how a game works, and totally overestimate how much work and modeling the simulator is actually doing, because they filled in the gaps with their imagination and preconceptions and assumptions, instead of realizing how many simplifications and shortcuts and illusions it actually used.

https://www.masterclass.com/classes/will-wright-teaches-game...

>There's a name for what Wright calls "the simulator effect" in the video: apophenia. There's a good GDC video on YouTube where Tynan Sylvester (the creator of RimWorld) talks about using this effect in game design.

https://en.wikipedia.org/wiki/Apophenia

>Apophenia (/æpoʊˈfiːniə/) is the tendency to mistakenly perceive connections and meaning between unrelated things. The term (German: Apophänie) was coined by psychiatrist Klaus Conrad in his 1958 publication on the beginning stages of schizophrenia. He defined it as "unmotivated seeing of connections [accompanied by] a specific feeling of abnormal meaningfulness". He described the early stages of delusional thought as self-referential, over-interpretations of actual sensory perceptions, as opposed to hallucinations.

RimWorld: Contrarian, Ridiculous, and Impossible Game Design Methods

https://www.youtube.com/watch?v=VdqhHKjepiE

5 game design tips from Sims creator Will Wright

https://www.youtube.com/watch?v=scS3f_YSYO0

>Tip 5: On world building. As you know by now, Will's approach to creating games is all about building a coherent and compelling player experience. His games are comprised of layered systems that engage players creatively, and lead to personalized, some times unexpected outcomes. In these types of games, players will often assume that the underlying system is smarter than it actually is. This happens because there's a strong mental model in place, guiding the game design, and enhancing the player's ability to imagine a coherent context that explains all the myriad details and dynamics happening within that game experience.

>Now let's apply this to your project: What mental model are you building, and what story are you causing to unfold between your player's ears? And how does the feature set in your game or product support that story? Once you start approaching your product design that way, you'll be set up to get your customers to buy into the microworld that you're building, and start to imagine that it's richer and more detailed than it actually is.

Also:

Will Wright on Designing User Interfaces to Simulation Games (1996) (2023 Video Update)

https://donhopkins.medium.com/designing-user-interfaces-to-s...