Ask HN: How to avoid over-engineering software design for future use cases?

I started out by writing unit tests for everything, which became my handhold and documentation for what the system originally did. Then I started reorganising the code into a more structured and more readable form, without changing the functionality, as proven by my unit tests. Then I started asking domain experts what exactly it should do, unit tests in hand, asking if these answers were correct. If they weren't, I changed the unit test, and then changed the code to match.

Surprisingly painless for something that by all reasonable standards was a terrible mess.

If I had to boil the first part down, I'd say you need to focus your engineering on the interface points - network protocols, schemas, API sets, and persistence formats immediately come to mind. It's a truism of the profession that those are the most expensive aspects of a system to change, and therefore they're the least likely to be mutable down the road.

That's really the easy part... the harder part is maintaining the team and self discipline to keep things simple. For better or for worse, engineering job placement (and oftentimes personal satisfaction) is highly resume-keyword driven. Organizations and people all tend to chase the next resume keyword on the assumption that it'll help them deliver more efficiently or get the next job placement or write the next blog post or whatever else. The net of this is that there's a very strong built in tendency for projects to veer in the direction of adding components, even before considering whether or not they're appropriate for use. So keeping your eye on actual, real project requirements over all else is both important, and can require convincing and political work throughout a team and organization.

An example of this conversation might look something like "This design is nice and simple but here are the possible scaling bottlenecks. If we need to scale X then what would be the path for that?" or "There's no reason to make this configurable now but if we needed to in the future how would we do that?"

Those sorts of questions shouldn't drive the design but if you're trying to decide between two designs that feel roughly as good as each other then it's a good way to tip the scale. I also find it a good way to introduce junior devs to thinking through those kinds of ideas.

One thing I had to learn to be careful with was to make sure it's really clear we're talking about future hypotheticals. I've had cases where the conversation about future design ideas ended with "Ok I'll go get started on that" and then we have to talk about how we weren't discussing real implementation changes, just hypotheticals about how the design might work in the future.

Any insights into how to do exactly this? Any rules of thumb or guidelines? Also what is not considered complex to some adds a cognitive burden to others.