Ask HN: How to avoid over-engineering software design for future use cases?

This by far the main reason to have people with decades of experience in engineering teams.

When I was young and fresh, I wanted to cater for all sorts of future possibilities at every turn. But what if things change this way? What if things change that way? I came to realise that when things change, unless the design change is trivial (allow use of database Y as well as Z, etc) then you probably haven't anticipated the way in which it is going to change anyway. Better to have clean, straightforward code than layers and layers of abstraction and indirection, to the point where it's hard to tell what's actually going on. You'll probably find that when change does come, your abstractions are an annoyance and a straight-jacket.

A junior engineer recently presented me with some completed work. There was a data schema change on one interface and an API version change on another. He had created a version of the microservice he was working on which could be configured to work on either the old or new version of each of these, with feature flags, switching functionality and maintaining the old and new model hierarchies for both. He viewed this as an achievement. While technically it was, the result was code bloat and unnecessary complexity. The API upgrade and schema change were happening at the same time, and would never be rolled back, his customisability was a net negative.

Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it. Don't build to requirements you can imagine, because the ones you don't will kick your ass.

Here's how I look at it.. when writing code, you need to run it to try it.

Often people refresh the browser or re-run their CLI program until their feature is finished.

But if you think about it, every "refresh to check if it works" is just a manual test. TDD is just making that manual test automated.

1. Write that test (that you'd anyway have to run manually)

2. Write code until test pass

3. Repeat 1 until done.

Code that aren't designed with a test-first mentality is often really hard to test and require complicated tools or need to mock the whole world.

For the examples you've mentioned:

- I'd unit tests the db service layer (I.e. functions to fetch from db, make sure schema is valid)

- I'd unit tests the various API queries (I.e. filtering, pagination, auth)

- At the controller level, I'd just unit test the business logic and data fetching part.

- Then I'd add a few E2E tests for the UI and user interactivity.

But if you think about it, any of these tests would have had to be run manually anyway. I.e. You'd probably have queried your API with various options and refreshed the page(s) a few time to make sure data was fetched correctly.

> engineers go extreme in designing things/code for future cases which are not yet known

They're afraid.

Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.

But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.

So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.

If there is an actual customer, try to get the scale of users/data the system is expecting. A system handling 1/1k/10M things every day will be quite different and need different solutions.

Problems arise when people reach for the Cool Tools and start building Webscale things that can handle 100M operations every second. ...but the customer only needs the system to handle 4 users who type in everything crap by hand. But hey, it has a clustered database and Kafka and Kubernetes and looks REALLY good on your Resumé.

When the scale is determined, I personally like the mantra of "Make it work first, then make it pretty". First build an MVP (or an ugly Viable Product), that proves your plan actually works, then you can iterate over it to make the implementation cleaner or faster or have better UX.

If you get stuck making everything super-generic and able to handle all possible cases, you'll spend time bikeshedding and never get anything deployed. Just Repeat Yourself with wild abandon and copy/paste stuff all over until your Gizmo actually works. You can spend time figuring out which parts are actually possible to make generic later.