Overengineering can kill a product • Hacker News Board

Re: Overengineering can kill a product

#71

Post by magicalhippo » Wed, Nov 24, 2021, 2:12 PM UTC

jrochkind1 wrote: post #33

scottious wrote: post #27

The insidious thing about over engineering is that it's usually committed by very experienced engineers. Experienced engineers rarely under-engineer, that tends to be fixed very early in one's career. As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way. Best real-world example: I inherited a project that was an…

OP suggests there's a hump in the curve of experience vs over-engineering; more experience correlates to over-engineering until it gets to a certain point, at which even more experience leads to less over-engineering. Is it true? I don't know, I think it matches my... experience. Part of it is that the "experience" needed to avoid over-engineering is helped if it's not just engineering experience, but domain experien…

Often I'll tend to over-engineer the initial solution, but as I think about all the complex edge cases that needs to be handled etc I often find myself asking "does the customer really need all this".

After discussing it with the customer, a simpler solution often emerges, where they change the requirements slightly allowing for a much simpler solution that might even solve the actual needs better.

While it's not just down to experience, I'd say it has a strong influence on being able to see beyond the given requirements towards a better outcome.

Re: Overengineering can kill a product

#72

Post by sokoloff » Wed, Nov 24, 2021, 2:12 PM UTC

somehnacct3757 wrote: post #60

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet aft…

With the benefit of hindsight, I think you can conclude that the system was over-engineered. If you’re trying to answer only “was it over-engineered?” I think you can answer it with hindsight.

If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.

Re: Overengineering can kill a product

#73

Post by noisy_boy » Wed, Nov 24, 2021, 2:16 PM UTC

somehnacct3757 wrote: post #60

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet aft…

My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you are wondering about those same things, trust me, there is a reason/background - its not because the people who did it that way were stupid".

Re: Overengineering can kill a product

#74

Post by pdenton » Wed, Nov 24, 2021, 2:19 PM UTC

Oh boy, does this ring a bell with me. I've already written 5575LOC according to cloc (and threw away 7449LOC in the process) and I'm far from finished writing the code to email the data from a contact form in PHP. But it ticks all the boxes!

It's all OOP

SOLID principles

99,6713% typed (according to psalm)

Purposely written to be unit-testable in PHPUnit strict mode (but no actual harness yet)

...I'll show myself out

Re: Overengineering can kill a product

#75

Post by noisy_boy » Wed, Nov 24, 2021, 2:23 PM UTC

abraxas wrote: post #44

yosito wrote: post #32

Six months ago I left a company that was working on an overengineered product. Even worse than it being overengineered was that it was also under documented. Working on anything was a pain, because the CTO wanted everything to follow his well thought out, and frankly very cleverly engineered design patterns, but he couldn't clearly communicate what those patterns were. And the entire company amounted to transforming…

Being ins a similar role myself, how do I ensure that engineers stay happy working on the project that we're working on? I'm finding myself actually doing the opposite of the CTO you mentioned and pushing them towards adopting more off-the-shelf components instead of maintaining homegrown stuff but I think I'm causing a degree of upheaval by doing this. Their justifications for push back however, often smell of sunke…

I am a developer and I always try to re-use a solution (preferably open source) instead of rolling my own (I'm happy to do the latter in case the need/situation demands but those cases are not that frequent). I also happen to enjoy devops (specifically CI/CD stuff and enjoy the integration aspects of connecting things to achieve a goal and see the entire pipeline producing the end-result). So I guess you'll have to find people that have the integrator mindset instead of those with NIH syndrome. At the same time, you do need some talented developers as well in case you do need to roll your own. It is a balance like most things.

Re: Overengineering can kill a product

#76

Post by mumblemumble » Wed, Nov 24, 2021, 2:27 PM UTC

somehnacct3757 wrote: post #60

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet aft…

It's a question of framing. Nothing is ever over- or under-engineered. It's over- or under-engineered for a purpose.

What that distinction recognizes is that it's possible for something to have been well-engineered at one time, while still being over-engineered today. Header files in C and C++ are an example of this phenomenon. They solved a very real problem with technical constraints from 40, 50 years ago, both in terms of computers' capabilities and compiler technology, but, since those problems don't exist anymore, they function as over-engineering.

Re: Overengineering can kill a product

#77

Post by noisy_boy » Wed, Nov 24, 2021, 2:29 PM UTC

bluGill wrote: post #51

lbriner wrote: post #50

As others have said, it is unecessary complexity. Over-engineering is an ambiguous term. For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all…

> use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and…

Also depends on expected changes to the no. of items. If you think it can shoot up, then you are better served by sticking to map/dictionary (and paying the minor penalty, assuming not in hot path) compared to starting with array for low-volume and then having to make code changes to map/dictionary to handle increased volume.

Re: Overengineering can kill a product

#78

Post by noisy_boy » Wed, Nov 24, 2021, 2:36 PM UTC

gampleman wrote: post #66

One thing that bugs me is the notion that "Software rewrites are something you should never do", which is a mantra so often repeated that it has acquired the status of self-evident truth, despite the only evidence being (usually) presented is an example of a web browser from 20 years ago! (Which incidentally spawned Mozilla, so not exactly a complete loss; especially from the POV of society rather than shareholders,…

Having dealt with a similar setup that basically had to be re-written, I totally agree. I would also like to add that if one senses themselves to be in a situation like this where the system is a messy build-up over years, try to resist adding things that are absolutely not essential. Otherwise, the guy doing the clean-up/re-write down the line may be forced to take not-so-clean approaches to cater to those non-essential bits mainly for backwards compatibility.

Re: Overengineering can kill a product

#79

Post by cfn » Wed, Nov 24, 2021, 2:41 PM UTC

gampleman wrote: post #66

One thing that bugs me is the notion that "Software rewrites are something you should never do", which is a mantra so often repeated that it has acquired the status of self-evident truth, despite the only evidence being (usually) presented is an example of a web browser from 20 years ago! (Which incidentally spawned Mozilla, so not exactly a complete loss; especially from the POV of society rather than shareholders,…

I suspect that the main issue with rewrites is that the users or product managers see it as an oportunity to add new features or redo old ones extensively. In the end the scope of the rewrite is no longer a rewrite but a new product that is incompatible with the original it was supposed to replace. I have seen this happen a couple of times. A straight rewrite for technical reasons and well defined scope does not suffer these issues.

Re: Overengineering can kill a product

#80

Post by mgkimsal » Wed, Nov 24, 2021, 2:42 PM UTC

noisy_boy wrote: post #73

somehnacct3757 wrote: post #60

I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written. Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet aft…

My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you ar…

Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'.

Examples

Having a 'user info' table with 190+ boolean columns for 'country' ('US','CA','DE','IT', etc) in case someone wants to indicate they're 'from' two countries.

Joining views on views of joined views which are themselves built on joins of other views is, likely, not a terribly sound data decision. (X: "It worked fine last year - you must have broken something." Me: "well... last year you had 450 users, and now there are 37000 users - this isn't a performant way of doing this". X: "Can't be true - the person who wrote it is a DBA")