Fixing under-engineered code vs. fixing over-engineered code • Hacker News Board

Re: Fixing under-engineered code vs. fixing over-engineered code

#31

Post by megameter » Thu, Aug 26, 2021, 6:20 PM UTC

fennecfoxen wrote: post #16

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

If we're sharing experience with underengineered code, let me have a go. At $BUSINESS we have a very successful marketplace that brings together buyers and sellers! We've recently IPOd. We get a lot of new items in and we have an internal page which is used by multiple full-time employees to approve new items, maintaining quality and defeating spammers. The code is written in HTML::Mason templates in Perl (which is b…

You did all that in a few months? That actually sounds like something that went quite smoothly. The time needed to engineer that kind of high-scale system at the outset is also going to enter the multi-month range, and with less background info to inform it. The phase-shifts of scaling are by no means easy, but they respond well to effortful, issue-by-issue grinding away at the problem.

The worst-case overengineered projects generally have organizational issues that preclude ever making concrete progress or result in a Juicero-style "why did you even make this" product. Those are the projects that burn people out.

Re: Fixing under-engineered code vs. fixing over-engineered code

#32

Post by p4bl0 » Thu, Aug 26, 2021, 6:22 PM UTC

xnx wrote: post #19

the_sleaze9 wrote: post #7

Earlier quoted context omitted.

The acronym YAGNI itself is flexible enough to handle this edge case accurately. The GO4 were truly visionary.

GO4?

Not sure but I suspect it would be the "gang of four", i.e., the authors of the canonical OO design pattern book.

Re: Fixing under-engineered code vs. fixing over-engineered code

#33

Post by SquishyPanda23 » Thu, Aug 26, 2021, 6:23 PM UTC

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

> If I get a task to "make the button blue" I'd rather do it in a repo where I need to grep around a little

Except it's not one button, it's 12 of them and they all use different UI elements with different names and syntax, and you can't change one of them because doing so would break some unrelated business critical class.

Re: Fixing under-engineered code vs. fixing over-engineered code

#34

Post by merrywhether » Thu, Aug 26, 2021, 6:26 PM UTC

hermitdev wrote: post #17

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

Your experience largely mirrors my own. At a previous employer, I had to make some changes to a process that was importing data from a vendor. Typical straight-forward ETL, right? Not even a lot of data, like 20-30 records daily. The process? Load the data from CSV to JSON, ship the JSON off to Azure. Pull the data back from Azure, check to see if it's been processed, apply the change on premise. If failed, reschedul…

To be slightly pedantic, one might call this situation over-architected rather than being purely over-engineered. I would agree that over-architected solutions (regardless of the internal engineering quality) are definitely hard to alter due to the many disconnected disparate parts, not to mention any human/political boundaries that have grown up around the implementation.

Following along that train of thought, under-architected solutions are often great to update because you get to make logical cleavages that are informed by time spent in actual production use, giving you a much better basis for any decisions.

Re: Fixing under-engineered code vs. fixing over-engineered code

#35

Post by pmkiwi » Thu, Aug 26, 2021, 6:27 PM UTC

hermitdev wrote: post #17

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

Your experience largely mirrors my own. At a previous employer, I had to make some changes to a process that was importing data from a vendor. Typical straight-forward ETL, right? Not even a lot of data, like 20-30 records daily. The process? Load the data from CSV to JSON, ship the JSON off to Azure. Pull the data back from Azure, check to see if it's been processed, apply the change on premise. If failed, reschedul…

This reminds me a colleague creating a Python class like this

import os

class Bla(): def __init__(self):

        self.var1 = os.env["VARIABLE1"]

        self.var2 = os.env["VARIABLE2"]

        # ...

        self.varN = os.env["VARIABLEN"]

    def get_variables(self):
        return self.var1, self.var2, ..., self.varN

They took 2 months to write an insanely complcated code to just copy few objects from an S3 bucket to another... And because the Lambda was timing out, they set the timeout to 15 minutes, leading to our Lambda costs skyrocketing because the function was failing/retrying all the time for some objects.

I was allowed to fix the timeout issue to save cost but I was forbidden to fix the code itself because our manager said "Well, it works so let's move on".

6 months later, on a Tuesday morning, bored, I decided it was enough so I rewrote this bloody Lambda in ~90 lines with a proper handler, retries, logging statements, etc ... in a couple of hours.

What did my manager say? "Good work but you should have taught them rather than doing it yourself".

Re: Fixing under-engineered code vs. fixing over-engineered code

#36

Post by albertzeyer » Thu, Aug 26, 2021, 6:39 PM UTC

So this article concludes that starting with a certain project size, over-engineered code is easier to work with than under-engineered code, and scales well when the projects grows (even "linear with the size").

But what this article is actually about is about entanglement of individual parts. Under-engineered implies spaghetti code with lots of entanglement. Over-engineered code does have the minimum needed amount of entanglement.

Of course this is right on the topic of entanglement.

But this is not so much my understanding of the terms over-engineered vs under-engineered.

I understand over-engineered as having too much abstractions. And those can turn out to be the wrong abstractions when they are actually used, which then requires more work to refactor this.

I understand under-engineered as having too little abstractions, and having build in some assumptions which might hold for one use case but not anymore for others. This can also require some effort to fix this.

In practice, this is also a continuum, and you might even have both things mixed.

And when actually writing some new code, it is often hard to know the right amount of complexity.

This is why the overall quality greatly improves when you do a couple of iterations of rewrites from scratch. Because you keep improving on just the right needed abstractions.

https://github.com/Droogans/unmaintainable-code

Re: Fixing under-engineered code vs. fixing over-engineered code

#37

Post by yongjik » Thu, Aug 26, 2021, 6:40 PM UTC

hermitdev wrote: post #17

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

Your experience largely mirrors my own. At a previous employer, I had to make some changes to a process that was importing data from a vendor. Typical straight-forward ETL, right? Not even a lot of data, like 20-30 records daily. The process? Load the data from CSV to JSON, ship the JSON off to Azure. Pull the data back from Azure, check to see if it's been processed, apply the change on premise. If failed, reschedul…

Amen. And the worst part? Can't even fix it easily, because they all depend on each other. Someone is getting the data in json format and depends on the filename being exactly YYYYMMDD in a specific directory of a specific server. Some horrible open source framework can only handle data at 200 requests/sec (and of course every row is a request) and you have a distributed asynchronous queue plus rate-limiter to make it work. Yada yada.

Re: Fixing under-engineered code vs. fixing over-engineered code

#38

Post by ellen364 » Thu, Aug 26, 2021, 6:42 PM UTC

ghiculescu wrote: post #3

This misses the point that overengineering takes longer to do, so it adds costs to the initial development process. Unsurprisingly there’s less cost later - you’ve already paid a lot of it!

That seems like the best case scenario — the over-engineering is costly upfront and worth it later on. But if there’s a problem in the over-engineering, e.g. mistaken assumptions, fixing it is another round of expense.

So over-engineering is “expensive now and hopefully cheap later”, while under-engineering is “cheap now and maybe expensive later”. In one scenario you get 1-2 rounds of expensive. In the other you only get 0-1 rounds of expensive.

With that in mind, under-engineering the first implementation might be the sensible default choice.

Re: Fixing under-engineered code vs. fixing over-engineered code

#39

Post by drran » Thu, Aug 26, 2021, 6:54 PM UTC

LeifCarrotson wrote: post #26

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

Over-engineered code has a lot of coupling where incidental equivalence may have been misapplied as fundamental sameness. This was discussed here on one of my favorite PLC programming blogs: https://www.contactandcoil.com/automation/industrial-automat... Copy and paste are great tools. Making all 6 buttons in a particular grid with the code: grid.AddNewButton(1, 1, "Thing 1", Color.White, Color.Black, onClick1()); gr…

Foreground and background color should be moved out to avoid repetition, e.g. BUTTONG_FG, BUTTON_BG. All other parameters are different, so no repetition.

Re: Fixing under-engineered code vs. fixing over-engineered code

#40

Post by awkward » Thu, Aug 26, 2021, 6:55 PM UTC

fennecfoxen wrote: post #16

awkward wrote: post #9

This is the opposite of my experience. Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases. Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project. If I get a ta…

If we're sharing experience with underengineered code, let me have a go. At $BUSINESS we have a very successful marketplace that brings together buyers and sellers! We've recently IPOd. We get a lot of new items in and we have an internal page which is used by multiple full-time employees to approve new items, maintaining quality and defeating spammers. The code is written in HTML::Mason templates in Perl (which is b…

LOL I'm very sorry that happened. None of this is scientific - over and under engineering in this thread are abstracted from actual results. I'm more than willing to admit that convoluting concerns between presentation and billing is nuclear waste grade underengineering.

For my example of overengineering, I was specifically working with a project where every web page on a public site was taking a couple seconds to load. A team had reimplemented npm using inheritance in docker. Because they had one library that just imported stuff (to populate the parent docker image), their webpack build was unable to distinguish between imported and unimported code, and was just packaging everything. 20 mb websites.