Live data from Hacker News

The collapse of complex software

nolanlawson.com

81–90 of 304 posts

Re: The collapse of complex software

#81
A legacy system exists: it’s big, it’s complex, and no one fully understands how it works. Architects are brought in to “fix” the system. They might wheel out a big whiteboard showing a lot of boxes and arrows pointing at other boxes, and inevitably, their solution is… to add more boxes and arrows. Nobody can subtract from the system; everyone just adds.

In my experience this is the only safe thing to do when you don't fully understand the system and cannot reason about it in a precise way. You create well defined abstractions over the messy parts, verify the abstractions, and write lots of tests. Eventually you can start deleting code. Although nobody ever ends up deleting code because they skip the "well defined" part and end up adding more indirection, afraid to delete code.

The only real way to tame complexity is to reason about it with precise, mathematically-defined abstractions. You can't invent Paxos using boxes and arrows on a white-board and prove the minimum quorum required without showing your work. Fortunately we have languages and systems that can automate a lot of the effort in checking our work as long as we can express it in formal language.

The other problem with this is that learning how to use these tools and techniques requires a lot of effort that isn't highly valued or rewarded in industry. Most businesses are fine with the trade off of errors and mistakes down the line if they're first to market. Ship it now, fix it later. We do a reasonably good job of preventing enough obvious errors with our tooling and processes. We can have software that obviously has no errors or software that has no obvious errors. Time and again we chose the latter because the former is too difficult and doesn't reward us with money now.

Re: The collapse of complex software

#83
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

Have you looked at Clojure and REPL/data driven development?

Re: The collapse of complex software

#85
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

This assumes that complex software problems are decomposable into discrete components defined and analyzable solely by their public interfaces. Complex software frequently cannot be modeled in this way -- it is what makes the software "complex". In reality, components have complex interactions far outside of what is captured in the component interfaces (see also: Hyrum's Law). Surfacing these implicit interactions at…

There are many embellished, hand wavy statements in this comment.

Could you give real world examples of:

  * "In reality, components have complex interactions far outside of what is captured in the component interfaces"
  * How template metaprogramming "precisely express[es] behavioral contracts... that are difficult to do in other languages" 

I think you contradict your poetry with "Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity". If _any_ non-trivial software system _can_ be reasoned about as a monolith then it is true there exists some model with sufficient simplicity that there are well-defined discrete components.

Re: The collapse of complex software

#86
post #80

I feel like it's a bit backwards. The software complexity increases because people lack the patience to tackle the existing complexity, or learn why things are complex, and prefer to believe that it can be simple, or get job done somehow, and in order to do that, they (somewhat egoistically) go their own way, which actually - barring occasional good insight - mostly increases the total complexity. The same goes for s…

I believe many software developers often tend to write software which has precisely the segree of complexity they can just barely manage. There is not many people who have the guts to build a simple straightforward step for step script. Software developers that fall into love with the beauty of their own abstractions can be a real problem.

Re: The collapse of complex software

#87
> engineers like complexity

Is that really true? If it's descriptive, then I'm not an engineer (and I may well not be). I don't like complexity at all, in any form. I want the least complex solution to any problem that will work. However, I've mostly given up on decomplexifying, because it's mostly a waste of time. Software becomes complex because getting something done fast, right now, "this time", because, "we're in a hurry this sprint" by duplicating code will always win over trying to modify something to be reusable. If you try to spend time fighting complexity, you won't just be fighting the complexity inherent in the existing system, you'll be fighting your coworkers who've given up on fighting complexity and just get their tickets into a done state.

Re: The collapse of complex software

#88
post #68

Earlier quoted context omitted.

> I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle. So in that view, you can't succeed at making software engineering alway…

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation. Frameworks approximately do this, but for specific domains: th…

> tackling more complex problems

This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.

However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.

> small team of engineers

It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.

As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.

> Frameworks

One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.

Re: The collapse of complex software

#89
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

Isn’t this the dream since year dot? The problem is the real world is full of leaky abstractions, with different layers evolving at different rates and massive adoption inertia.

Re: The collapse of complex software

#90
post #80

I feel like it's a bit backwards. The software complexity increases because people lack the patience to tackle the existing complexity, or learn why things are complex, and prefer to believe that it can be simple, or get job done somehow, and in order to do that, they (somewhat egoistically) go their own way, which actually - barring occasional good insight - mostly increases the total complexity. The same goes for s…

> they (somewhat egoistically) go their own way,

That's not necessarily all ego talking. If you have a task to accomplish and you're going to need to spend a week looking at it just to get a handle on how to start getting it done the simplest overall way possible, your boss is just going to insist you do it the "quick and dirty" way "this one time" (every time). As Dilbert says, "our boss can't judge the quality of our work, but he knows when it's late". This is an inviolate law of software development: quick always beats good. Get on board, or you'll be replaced by somebody who is.

Post reply on HN