Live data from Hacker News

The collapse of complex software

nolanlawson.com

111–120 of 304 posts

Re: The collapse of complex software

#111

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

I'm curious how big of a factor a consistent core team is to having long-term sustainable and stable software. When companies have people rotating on projects constantly and the original vision and design principles of the product are lost or changed, I can only imagine that contributes to the problem.

Re: The collapse of complex software

#113
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

Can you elaborate and/or give examples? I read your comment and I can only see vague pseudo-profound statements.

Re: The collapse of complex software

#115
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

A very worthy, laudable goal. In addition to some of the other good responses here, my reason for thinking it can’t be solved with better tools alone is that, contrary to what the article suggests, engineering & tools alone are not the primary cause of our complexity, customers and consumers and management are. Business and the need to keep making money is what causes software to need to evolve and change, to support new features that weren’t planned into the architecture, and to develop software in shorter timeframes than is necessary for clean solutions. This is what I’ve seen in a few decades of professional programming, that programmers are perfectly capable of solving hierarchical and cognitively complex problems. Customers or salespeople ask engineering to implement strange inconsistent features, people aren’t happy with your software because your competitor does X, and we’re always walking a balance between time and resources to meet those demands, and there’s never enough of either. A lot of software business, for better or worse, is also driven by fads, so need complexity inducing maintenance just to not feel old and cruddy.

The other reason complexity happens is dependencies and DRY thinking, which is often good, but dependencies, centralization of code, and sharing of code are also a risk. Avoiding repetition is what causes abstractions to grow. Using other people’s libraries means that there are complex interactions you don’t understand. Most of the time it’s all fine, but occasionally sometimes it’s not. This trade is made consciously, for the reason that it’s much faster to develop your app using existing libraries, and not reinvent all wheels, and nearly everyone is doing it. The reason that tooling is unlikely to solve this part of the problem is that failures are so often caused by incorrect expectations - someone using a library hoping that it will do X when it only does Y. Even in theory, better tools could only tell us low-level information about what a library does, but I don’t think it’s possible for tools to tell you at a high level what a library can’t do.

Re: The collapse of complex software

#117
post #28

I highly recommend Rich Hickey's "Simple Made Easy" talk as a way to think about software complexity. We all know that complex systems are hard to work with, but I've found his presentation very useful in understanding the nature of complexity. Understanding complexity helps being able to avoid or minimize it. For example, easy is not the same as simple, and simple changes to one part of a system can introduce great…

I've watched that video at least three times, but I still can't articulate the exact distinction he is making between simple and easy :/

A for loop is easy: everyone can conceptualize doing a process N times.

A map is not as easy of a concept in some ways (it's more abstract, you are taking N elements of X to N elements of Y). But a map is simpler than a for loop, because there is no state between steps: it's completely symmetrical.

The for-loop "complects" each iteration together, even if it's as little as incrementing an integer. It's a side effect, however trivial.

To kick it up a notch, now imagine a 2-D loop. Now we have 2 counters. If it's a 2-D array, each addressing operation interacts with 2 counters. But map just needs a second map chained on, and each map is completely "unaware" of the other map. You could map over N dimensions and each map is symmetrical, while the for loop needs an ever increasing number of counters. This slight difference in complexity shows how complexity compounds with other complexity.

Re: The collapse of complex software

#118

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

> Things like the linux kernel, all the GNU utils, emacs, vim, apache, nginx, postgres. The list goes on. We know how to write software that lasts for decades, continuously improves, and doesn't need constant re-writes.

At least half of the items on that list are still here purely on momentum and reputation from the time there were no alternatives, not because they were really that marvelous.

I use and even enjoy some of them, but they're far from great engineering.

Re: The collapse of complex software

#119

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

I think nginx is a rewrite of apache, just as busybox is a rewrite of coreutils. Linux tends to rewrite itself but just keep the userspace ABI.

Re: The collapse of complex software

#120

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

Allowing projects to fail is important too, it just happens that complexity plays a big role in these failures.
Post reply on HN