Complexity Explained

Progress.

I got clients via word of mouth and then worked with them to have a more structured & systematic way to think about SEO and online growth.

Now I get most of my clients via the book. And I already start with them from a systematic baseline.

------

Or think of it the other way, imagine getting contracted to create a webapp and the first thing that you need to do is to explain how the internet works.

Vs. Getting contracted and starting to talk about the advantages/disadvantages of different frameworks and app paradigms.

I've learned to think of understanding as a form of compression. The more recurring patterns there are in a system, the better we can understand them. The fewer such recurring patterns in a system, the harder it is to understand them. Fortunately for us, a lot of things in nature are recurrent.

Complexity is not just about these larger features, although they are certainly manifestations of complexity. So like a mechanical watch is complex, but it is complex in a very different way than weight gain or weight loss is.

Nevertheless yes, this is the basic idea. Systems theory speaks about how you have substances flowing between different buffers or containers, as well as feedback loops that look at the buffers and adjust new flows based on those buffers. Or at least that is one of the most universal models. A nice general set of theorems about this are the fixed point theorems, which say that in a very large set of circumstances such systems find themselves in a “fixed point,” a set of buffer quantities and flows that is self-sustaining and usually healing from small perturbation. This is why when you stop dieting you gain back the weight; the same causes lead to the same effects.

The complexity of systems theory in particular comes from a few different properties worth thinking about:

(a) not being able to predict the whole from the parts and thus not knowing how to change the system to facilitate the overall behavior change [in other words “if dieting doesn't work, what does?]

(b) metastability transitions where there is enough random variation to allow for spontaneous reorganization between multiple fixed points [in other words if I get skinny in the Netherlands and fat in the US, then maybe American pandemic-me is skinny but Dutch pandemic-me is fat—as a corollary there is no such thing as “best practices” really, the same “causes” do not produce the same effects: what is the difference with the above?]

(c) when you do manage to create change, it is typically by thinking wishfully about what the system should do and then driving it to produce that “at whatever cost” and initially some “bottlenecks” emerge which appear to be resistant to the desired change, which you want to think is bad, but what actually starts to happen is that everything else about the system re-architects to reinforce that “bad” thing and that makes it okay. [In other words the journey from fat to skinny may seem to be limited at first by my love of food and joyous hedonistic eating... if that's the main bottleneck then unlike a diet’s plan to make me deny myself, my healthy relationship with food will actually feature more of that, indeed the reason that I was fat was that I did not love food enough and with delectatio morosa consumed whole boxes of cookies not for my truest enjoyment of them but because the pleasure of even a few cookies was always denied to me. If you read Marie Kondo's book you may be struck that her diagnosis of mess is that you have too much stuff and her prescription for having too much stuff is perversely not to love your stuff less and be more stoic but to actually love your stuff more. Ecclesiastes seems to come to a similar conclusion.]

Yeah, I think those are the most outrageous parts of dealing with systems which makes me astonished by their “complexity.” I often find that if I ask “why can't we do that,” I get some response “because of such-and-so” and then I surprise my interlocutor by saying “but what if I accept the such-and-so?” and, we eventually arrive at a whole different self-consistent system.

One example of this in a tech context since I feel I have overmilked the health context: “What if we don't have merge conflicts?” “psh, yeah right, you're never gonna get rid of merge conflicts.” “ok but pretend I am dumb, tell me why?” “because people modify the same code in different ways and you have a conflict!” “so if I could make them take turns then there would be no conflict?” “well you can't make us all take turns on the codebase! That would mean only one person is allowed to work at any given time!” “Ah I see your point, but what if it's more like a Google Doc where everyone sees other peoples' changes in realtime, that sequences the edits so that they never have merge conflicts.” “but then we couldn't do code reviews properly!” “why does that matter?” “because when I am setting out to rewrite the auth logic, I put the thing into a half-finished state! The code review protects those failing states from getting to prod.” “Ok, so is there a way to develop without failing states?” “Well, kind of. Like I can write `if (false)` and then write breaking code. But if our codebase were littered with those I would never be sure what I needed to re-enable.” “So there's no way to define your own version of true and false for this thing you're working on?” “I mean, I guess a boolean variable. But that won't work either.” “why not?” “if you have 10 of those then you have 1,024 different states that the system can be in, it becomes a nightmare.” “and you can't remove them when a developer is done?” “not without proper code review! That's what I am saying!!” “okay but what if you then have both mini review and proper review, mini reviews just check to make sure that everything is behind a feature toggle, very quick approval-at-a-glance of work in progress: then bigger reviews happen when you instate the new code path and delete the old one. We could detect those mass deletions, yeah?” “I mean, yeah, but, I mean, this is just not how it's done. Nobody develops code like this.” “Well, can we try it as an experiment for a month and see if we like never having merge failures more than we hate doing things a new way?”

Any recommendations for books on the subject? I read Simply Complexity and hungry for more!

Complexity theory is fascinating in itself (talking about how efficiently solvable or complex a problem is in an absolute, mathematical way), but also can spur a lot of research. As an example, integer factoring is strongly believed to be hard for classical computers (in a well defined complexity theory sense). When Peter Shor presented an efficient quantum algorithm for integer factoring in 1994 it was a breakthrough for the field and triggered huge interest into the field of quantum computing and quantum alogrithms. Serious widespread research into quantum computing more or less started then. Quantum computing is estimated (by BCG) to create a market value of 450-850 billion $ by 2050 (mainly in chemistry, optimisation and cryptography).

So, the academically interesting aspect aside, it can spur research for useful things, it can act as a guide to what is worth / not worth researching (if something is proven to be in a "hard" complexity class, then you won't find any efficient algorithm to solve the problem, ever (assuming P =/= NP).) and is generally a great framework for algorithms research. And evidently, algorithms create a lot of value (good and bad, unfortunately).

It's just very much "under the hood". 100 years ago people could have also, rightly, said "how does quantum theory benefit people, what's the value in that". Yet, without the fundamental research in quantum theory we would not have modern technology. Understanding the behaviour of electrons in materials on an atomic level requires quantum mechanics.