Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Note on Essential Complexity

olano.dev

Post reply on HN
Print view View original

21–30 of 51 posts

Re: A Note on Essential Complexity

#21

Post by TeMPOraL »

tristramb wrote: post #19
smallstepforman wrote: post #11

Complexity builds rockets with thrust vectoring and lands modules on the moon, simplicity is good for fire crackers. Once you scale past simple prototypes, you need performance and new features. And the architecture stops being simple and complexity eventually creeps in. I’ve implented 4 iterations of a product from scratch, and eventually they all get complex, even though each one started out with the goal of being…

The vast complexity of the 21st Century has still not managed to land a single person on the Moon.

That's not on technology, that's on people holding the purse strings only caring about growing their purses. Capitalism grew up, and is a boring old fart now.

Re: A Note on Essential Complexity

#23

Post by silon42 »

boznz wrote: post #2

Humans simply cannot keep the whole stack for a complex system in memory, that is why we abstract layers with APIs etc and generally specialize on one layer only. My (Sci-Fi) book postulated that an AGI (a real AGI, after all it was Sci-Fi) would simply discard everything the humans wrote and rewrite the complete stack (including later on the Hardware and ISA) in machine code without anything unnecessary for the task…

> Humans simply cannot keep the whole stack for a complex system in memory

But IMO this necessary to fully debug it, unless you can prevent all leaky abstractions.

Re: A Note on Essential Complexity

#24

Post by podgorniy »

jt2190 wrote: post #7

I feel like this article over-complicates things by using a strange definition of “essential complexity”: That if a user says that something is necessary, than that is essential complexity. Personally I never assume that the user has distilled a problem down to its essence. Regardless, my process looks a lot like what the author recommends: Question assumptions, propose alternatives, eliminate work, etc.

Essential complexity is what program must model based on user needs. Modelling is done for the end user. That's why author mentioned "user says". It's not like "user understands how to describe what they need" rather "we model what user needs". Users will often ask for impossible or contradicting features and can't understand that without our help. I agree that average user will need a lot of help and work to define what they need/want.

Practical example of my abstract description.

User does not want cdn-cached, region-distributed, lambda-backed resizing and validating image services, usage of the latest image compressing formats with fallback for the older browsers with performance dashboats of speed of loading. User wants to see their profile photo on profile page. Showing that photo is an essential complexity as it comes from the end user needs.

Re: A Note on Essential Complexity

#25

Post by JonChesterfield »

kashyapc wrote: post #4

A well-written article. On a similar theme of essential vs. accidental complexity: FWIW, I summarised a talk gave by Paolo Bonzini (a Linux/KVM/QEMU maintainer) here in the past: https://lwn.net/Articles/872321/ -- A QEMU case study in grappling with software complexity Although it was in context of QEMU, the lessons from it can be applied to many other projects.

Thanks for this. Lessons from reality hit harder than from theory. I'm amused to learn that the distinction between essential and accidental was first described by Aristotle given that millennia later people are still struggling with the notion.

Re: A Note on Essential Complexity

#26

Post by andrelaszlo »

I find it interesting that we often think of "the product" and "the organization" as two separate things.

A common counter-example is when we start seeing (some) bugs and incidents as symptoms of organizational issues. This seems to get more visible with experience (less experience: "I screwed up" vs more experience: "this happened in a certain context, how can we prevent it from happening again?").

Conway's law is a brilliant observation but I think it's a much wider pattern: we build organizations and software at the same time, and they're inextricably linked.

In parallel to software design patterns, we have a bunch of organizational patterns and anti-patterns as well. SRE, agile methods, DevOps. Silos, CYA-ism, 10x developers, ...

Some examples of properties of an organization that directly impacts the properties of software:

- How well the organization embraces risk [0]

- Psychological safety[1]

- Hiring practices and diversity

- The ability of the individuals, teams, departments, etc in an organization to align and work towards the same goal

I'd love to hear some anecdotes of how code and products are directly impacted by these.

0: Risk-driven architecture seems like one attempt at making this explicit https://www.georgefairbanks.com/book/

1: https://hbr.org/2023/02/what-is-psychological-safety

Re: A Note on Essential Complexity

#27

Post by worstspotgain »

There's a lot to like in this article, but I would go a step further with its meta-abstractions. At complexity 0, there is no software to be written. In essence, complexity and the software itself are closely related, akin to the relationship between compression and AI. The article's thesis is that minimizing complexity is what engineering is about. That means we're kind of minimizing the total amount of software we have to write and maintain. Makes sense, so far so good.

Next up is architecture. More architecture generally means more complexity in the short term and less in the long term. But that's assuming we know what the long term will look like, which we can often only predict. If we under-provision architecture, there's a good chance we'll get buried in technical debt later on (though by then we may have switched gigs.) Enter the usual Agile arguments favoring the short term.

Last but not least, path-dependency. I think this is where the best designers shine. You might not need all the architecture right away, but you do need to steer clear of dead-ends. These are the bad quick-design decisions that bite you in the rear when the architecture eventually gets refactored in. An extreme example might be the lack of a version field.

IMO, awesome designers are great at predicting where the architecture will be and at forward-provisioning it. They don't need to build the freeway through the middle of town. They'll just plan a park here and a parking lot there, guessing the right spots.

Re: A Note on Essential Complexity

#28

Post by rob74 »

> Now, I want to challenge the notion that essential complexity is irreducible.

> The complexity is accidental, so we can remove it.

Yeah, never mind essential complexity, let's start by removing all the accidental complexity in today's standard web development stack (my estimate: 90% accidental, 10% essential)! Easy, right?

Re: A Note on Essential Complexity

#29

Post by wcrossbow »

It is kind of obvious but I think it is worth mentioning out loud regarding whether how to classify something as essential or accidental complexity is that where it ends up falling is only easy to tell in hindsight. Recently I finally realized, after working for almost 3 years in a problem, that it could all be framed as graph problem. This simple change of frame dissolved hundreds of lines of code into `reduce(find_shortest_path(graph, start, end), some_op)` style single lines. Now I understood that what I was previously doing was manually crafting paths through this informally defined graph across nodes I knew had connections. The ratio of comments to code was about 5:1 to make sense of it. Before that I believed all of it was true essential complexity but looking through the right glasses it really wasn't. The lessons for me (some of which I seem to have to constantly relearn) where:

1. It may only look complex because you are not thinking about it in the right way. 2. The right data structure/abstraction has the power to turn the worst spaghetti into a shining beacon of simplicity.

An important corollary is that if it looks complicated is probably because you don't understand it yet. Tautological yes, true, also :D

Not trusting myself on what is essential complexity and insisting on hitting my head against the wall until I found a solution has served me well many times in my life and career.

Re: A Note on Essential Complexity

#30

Post by CRConrad »

TeMPOraL wrote: post #21
tristramb wrote: post #19

Earlier quoted context omitted.

The vast complexity of the 21st Century has still not managed to land a single person on the Moon.

That's not on technology, that's on people holding the purse strings only caring about growing their purses. Capitalism grew up, and is a boring old fart now.

Capitalism supplanted its father, Mercantilism, in the eighteenth or at the very latest the nineteenth century. It was "an old fart" well before WW2.
Post reply on HN