Ask HN: How to avoid over-engineering software design for future use cases? • Hacker News Board

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#251

Post by mcv » Thu, Jun 25, 2020, 6:59 AM UTC

agloeregrets wrote: post #162

mcv wrote: post #121

Earlier quoted context omitted.

End-to-end tests are more overhead than unit tests. End-to-end tests are also not great for testing the many edge cases that you are likely to mess up if you refactor carelessly. I also don't see how a robust type really helps there. It might even be part of the thing that needs to be refactored. Besides, many languages don't have a very robust type system. End-to-end tests and type systems certainly have their uses,…

Unit tests are too. What the comment you are replying to is referencing is Integration tests, the type that instead of working on a “this class returns this” instead works on “component A calls Component B to do something, are the two still speaking the same language and expectations?”

There's three levels of testing: unit, integration, and end-to-end.

End-to-end (e2e) tests the entire stack: deploy the site with database and all, run a script that visits a page, clicks and stuff, and checks if the right things become visible.

Unit tests are the other extreme: you take a single unit of code and test whether it does what it's supposed to, while mocking all communication with other units of code.

Integration tests are in between those two extremes, and probably the least understood as a result (at least by me). They look like unit tests, and don't generally test the UI, but they don't mock the other components, and ideally also set up a database to test against.

I think there's a bit of overlap between unit and integration tests; a sloppy unit test where you don't mock (some) other components but treat them as part of the unit you're testing, start to look like integration tests at some point. If you want a clear demarcation, I think you might consider a database connection essential to count as integration test.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#252

Post by peppery-idiot » Thu, Jun 25, 2020, 8:52 AM UTC

donw wrote: post #64

The short answer is: don't design for future use-cases. Period. Instead, only build what you need to solve the problems you have in-hand, but do so in such a way that your software and your systems are as easy to change as possible in the future. Because you very, very rarely know what your future use-cases really are. Markets change over time, and your understanding of the market will also shift, both because some o…

I disagree on Clean Code. It puts too much importance on superficial "5 easy steps" points that put too much priority on the wrong things. Reading quality open source code is a better way of getting that understanding IMO.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#253

Post by nate » Thu, Jun 25, 2020, 11:41 AM UTC

I think most of the time this is just Parkinson's Law playing out: "work expands so as to fill the time available for its completion"

Things get over-engineered because too much time is allowed for the task.

The forcing function that's helped me is simply giving me and my teams small but meaningful time boxes to get a feature done. Sort of like "sprints", but with more teeth. E.g. An important feature is going to get announced in a newsletter ever 2 weeks. Sprints too often don't seem to focus on the shipping to customers part.

So we focus on shipping the smallest thing that could possibly work in an arbitrary time box. You know users need X. So you make X or X' or X'' - some version or whittled down version of X that'll relieve user pain. The time box does works wonders from keeping over complexifying from getting out of control.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#254

Post by Nursie » Thu, Jun 25, 2020, 1:20 PM UTC

qes wrote: post #241

Nursie wrote: post #14

When I was young and fresh, I wanted to cater for all sorts of future possibilities at every turn. But what if things change this way? What if things change that way? I came to realise that when things change, unless the design change is trivial (allow use of database Y as well as Z, etc) then you probably haven't anticipated the way in which it is going to change anyway. Better to have clean, straightforward code th…

> The API upgrade and schema change were happening at the same time Atomically? What if your DDL times out? What you describe is, in my experience, extremely standard process for rolling out breaking changes (you do of course remove the switch and old version support after everything is rolled out).

> Atomically? What if your DDL times out?

Effectively yes, during a defined outage period (don't ask, this is finance), on any error anywhere in the platform during rollout, the entire platform would be reset to its prior state.

> What you describe is, in my experience, extremely standard process for rolling out breaking changes

But entirely unnecessary here. There was no requirement for a single version of the microservice to be able to address multiple versions of either dependency. It was a net negative to have that support, added significantly to the software complexity and the raw LoC.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#255

Post by valand » Thu, Jun 25, 2020, 4:42 PM UTC

mason55 wrote: post #50

JoeAltmaier wrote: post #31

Earlier quoted context omitted.

Yet 'spare and clean' can mean breaking the problem down into primitives. And primitives can be reusable. I agree, contriving baroque features and APIs are usually wasted time. But planning simple basic operations that others are built upon is good sense. When possible.

Yeah we call them "building blocks." Instead of building specific individual features we create building blocks that we can use to implement features. The key difference is that by exposing the building blocks to customers they can use the building blocks in ways that we didn't imagine. The most important reason for us to do this is so that our customers can always accomplish what they need, even if it's somewhat man…

I found out about primitives/building blocks after 2 years of dealing with foreign discipline,

but I have problem sharing it with people because I just don't the credential.

Is there a well-known book/resource that I can share with people about this?

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#256

Post by ChicagoDave » Sat, Jun 27, 2020, 3:02 AM UTC

Wilem82 wrote: post #165

ChicagoDave wrote: post #19

It’s hard to flip your brain, but abstractions are bad. Copying code for different business purposes is good. Simple patterns are vastly better than complex frameworks, even if you think it improves unit or integration testing.

> abstractions are bad They're as good as you make them. Abstractions aren't bad by themselves. > Copying code for different business purposes is good. Only when it makes sense, ie when you can't make a good abstraction.

Nope. Different domains shouldn’t share code or data. One domain may have a customer record with address info and another domain may have a customer record with past purchases.

Abstractions end up wiring things together that blur business rules.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#257

Post by aabbcc1241 » Tue, Jun 30, 2020, 3:43 AM UTC

I usually take YAGNI further and avoid using database (sql or nosql) at all (at least in the beginning)

My typical approach is to log all the commands to the file(s), and keep the system state in memory. When the server restarts, it simply replay most of the commands (skipping those intended to cause side effect)

This is like simplified event sourcing, or command sourcing.

I'm not relaying on any framework to build the backend, simple library handing rest and websocket API is enough.

I'll open source it soon but it's mostly as simple as it sounds, with some optimization to reduce disk space consumption after they becomes a problem in practice.

Re: Ask HN: How to avoid over-engineering software design for future use cases?

#258

Post by collyw » Tue, Jun 30, 2020, 9:58 AM UTC

kevstev wrote: post #211

collyw wrote: post #189

Earlier quoted context omitted.

One thing I notice is that software is becoming more brute force and less about "design" or "architecture" quality. Good abstractions at the correct level and the correct data modelling can make the code around it very simple. Instead of nice designs that are easy to reason about, we have lots of unit tests to "prove" that it works. Instead of making a system reliable and trap errors properly, we use Kubernetes to la…

This is about cost optimization to some extent. Writing perfect bug-free code is nearly impossible and expensive. Hardware is going to fail, network links can get flakey, packets can get lost. Accepting that failure is going to happen no matter what and building for resiliency to it is a far better approach. Unit tests are along the same line- its been measured that fixing a bug in production is 10x more expensive th…

I am not arguing that unit tests are not valuable for the reasons that you have given, just that they seem to be used in place of decent abstractions these days. I have worked on well abstracted code without unit test and badly abstracted code with unit tests. The former is far easier to work with and bugs take a lot less effort to fix - that translates to costs in developer time.

> This is about cost optimization to some extent.

Is all the complexity and developer time that is required with Kubernetes really cost effective? We added it at my work and it took around 6 months of developer time from what I could tell (it wasn't me who set it up). Now we have to debug it and none of our team is an expert, so lots of guesswork and googling. Using number of small pods has given us a lot more problems than one big machine. If you are running out of memory because of a large upload, restarting a pod when it crashes doesn't help you.

Kubernetes has lowered our cloud costs and given us high availability, but it has increased development time in a number of ways. Maybe as we get more experience it will get better, but so far I would not say costs have been optimized as a result.