Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Note on Essential Complexity

olano.dev

Post reply on HN
Print view View original

11–20 of 51 posts

Re: A Note on Essential Complexity

#11

Post by smallstepforman »

Complexity builds rockets with thrust vectoring and lands modules on the moon, simplicity is good for fire crackers.

Once you scale past simple prototypes, you need performance and new features. And the architecture stops being simple and complexity eventually creeps in.

I’ve implented 4 iterations of a product from scratch, and eventually they all get complex, even though each one started out with the goal of being simpler than the previous iteration. Yes, iteration #4 is more complex than #1, but it is more performant.

In parallel I’m building a new house, and each iteration of the plans is more complex. You try to manage compromises. You take 2 steps forward, one back. Which way do windows face, can an older person navigate, is there enough storage space, cost, esthetics, where does a dirty dog enter, where is the chimney for preppers, driveway and orchard, septics and wells, drainage and water collection, guest rooms and hot tubs, all on a budget … Simple wont do.

Re: A Note on Essential Complexity

#12

Post by w10-1 »

Complexity is just one of the many kinds of requirements. And yes, requirements (despite their name) can be somewhat fungible. And if you keep going, you'll think of yourself as a forcing function for humanizing humanity.

For me the best perspective is evolution: what system will it be best for those handling my consequences to have? And the more concrete, the better.

So, not essential complexity, but essential.

Re: A Note on Essential Complexity

#13

Post by 082349872349872 »

karmakaze wrote: post #3
boznz wrote: post #2

Humans simply cannot keep the whole stack for a complex system in memory, that is why we abstract layers with APIs etc and generally specialize on one layer only. My (Sci-Fi) book postulated that an AGI (a real AGI, after all it was Sci-Fi) would simply discard everything the humans wrote and rewrite the complete stack (including later on the Hardware and ISA) in machine code without anything unnecessary for the task…

OTOH a lot of the abstractions are created because problems aren't solved coherently but rather partitioned into organizational structures adding incidental complexity. The worst are -services that seem to always do too much/too little or have interface designs that suit itself than needs of clients/consumers. I spend a fair amount of time decomplexifying the org out of architectures when we realize that performance…

> there's the data that's read, the rules applied, and the data that's written

and each rule applied can do one of three basic things: https://news.ycombinator.com/item?id=32498382

I like to collapse layers that are doing the "same kind" of thing according to the taxonomy above.

Re: A Note on Essential Complexity

#14

Post by philipswood »

boznz wrote: post #2

Humans simply cannot keep the whole stack for a complex system in memory, that is why we abstract layers with APIs etc and generally specialize on one layer only. My (Sci-Fi) book postulated that an AGI (a real AGI, after all it was Sci-Fi) would simply discard everything the humans wrote and rewrite the complete stack (including later on the Hardware and ISA) in machine code without anything unnecessary for the task…

I wish we would refactor our stack!

Each level throws away a lot of the effort of the preceding layer.

E.g. after heroic engineering for the OS and memory management systems to present the illusory abstraction of a flat memory address space to the runtime, it almost immediately chops it up into smaller independent "packets" that don't interact directly.

Or the whole storage memory slow cache fast cache register divide.

I think we should be building processors that are made of tens of millions of very simple processors, each with their own memory and with dedicated "network on chip"-like communication electronics. These processors would directly map to language structures like objects/structs/functions.

Essentially flattening the stack so that there is just one level between language and silicon.

Re: A Note on Essential Complexity

#15

Post by danybittel »

smallstepforman wrote: post #11

Complexity builds rockets with thrust vectoring and lands modules on the moon, simplicity is good for fire crackers. Once you scale past simple prototypes, you need performance and new features. And the architecture stops being simple and complexity eventually creeps in. I’ve implented 4 iterations of a product from scratch, and eventually they all get complex, even though each one started out with the goal of being…

I'd say these are essential complexities, they are features necessary for the client. Accidental complexity would be, if you for example, assigned the work on your new house to different "teams". Then the "guest room" team also build a drainage, or used a prebuilt drainage, not connected to the drainage the other team built.

To quote a quote from the article: "In my experience most of the complexities which are encountered in systems work are symptoms of organizational malfunctions."

Re: A Note on Essential Complexity

#16

Post by rramadass »

Good Article on a very important topic.

Generally, Problem/Systems/Requirements Analysis should be done Top-Down but Solution Design/Implementation should be done Bottom-Up. This approach is key to managing Essential Complexity.

The Analysis starts with the "system-as-a-whole" and uses the scientific reductionist approach to figure out the constituent sub-modules and their interactions. This is applied recursively to sub-modules until we choose to stop at a desired level. This identifies the essential complexity at the module level (eg. need for DSP processing algorithms on certain media RTP/RTCP data flows). But in the process of breaking down the system we also will have identified the essential complexity in the interactions between modules (eg. the SIP/etc. protocol module responsible for setting up the above media data flows) and cross-cutting all modules (eg. performance, logging, exception-handling). So while constructing the solution we now can take care of all the identified essential complexities involved by spreading our functional modules over a layering of policy and mechanism modules (see https://en.wikipedia.org/wiki/Separation_of_mechanism_and_po... and https://en.wikipedia.org/wiki/Separation_of_concerns). This distribution of Function, Policy and Mechanism is what constitutes the Architecture of the System.

Re: A Note on Essential Complexity

#17

Post by boznz »

metadat wrote: post #6
boznz wrote: post #2

Humans simply cannot keep the whole stack for a complex system in memory, that is why we abstract layers with APIs etc and generally specialize on one layer only. My (Sci-Fi) book postulated that an AGI (a real AGI, after all it was Sci-Fi) would simply discard everything the humans wrote and rewrite the complete stack (including later on the Hardware and ISA) in machine code without anything unnecessary for the task…

I checked out your book link https://rodyne.com/?p=1252 >, is there a preview chapter or two somewhere? That's cool you wrote a compelling sci-fi but my list is competitive :)

I doubt the first two or three chapters will do it justice. I have put a free to download link on the bottom of that page for the complete book as an ePub, I will keep the link valid for a few days. Enjoy.

Re: A Note on Essential Complexity

#18

Post by langsoul-com »

smallstepforman wrote: post #11

Complexity builds rockets with thrust vectoring and lands modules on the moon, simplicity is good for fire crackers. Once you scale past simple prototypes, you need performance and new features. And the architecture stops being simple and complexity eventually creeps in. I’ve implented 4 iterations of a product from scratch, and eventually they all get complex, even though each one started out with the goal of being…

You want to start with simple, because simple will become complex overtime. Starting with complicated means it'd only become more complicated.

Simple is also relative though. No point preparing for Google scale, when there's less than 10 users a month. But, if the objective is something like discord, real time comms, then it'd be simpler to start with the correct language and framework for that use case.

Re: A Note on Essential Complexity

#19

Post by tristramb »

smallstepforman wrote: post #11

Complexity builds rockets with thrust vectoring and lands modules on the moon, simplicity is good for fire crackers. Once you scale past simple prototypes, you need performance and new features. And the architecture stops being simple and complexity eventually creeps in. I’ve implented 4 iterations of a product from scratch, and eventually they all get complex, even though each one started out with the goal of being…

The vast complexity of the 21st Century has still not managed to land a single person on the Moon.
Post reply on HN