Live data from Hacker News

The collapse of complex software

nolanlawson.com

211–220 of 304 posts

Re: The collapse of complex software

#211
> One thing working in complexity’s favor, though, is that engineers like complexity.

No, absolutely not for me.

I want simplicity.

The problem is that our expectations for software systems is so vastly more complex than it ever was in the past. It always grows more complex. The needs and wishes only increase.

Yes there are some developers who like building complex structures. Admittedly it can be fun. But eventually we reach a point where it's too much for one person to keep up with. Then the challenge shifts from complex software systems to complex knowledge and management systems.

It is theoretically possible to have an infinitely complex system which is managed by a perfect, infinitely complex organization. In practice it doesn't work, for numerous reasons.

One reason is that business drives software development, and business is seeking profit. So business doesn't want documentation or complexity management; it wants features. That means a lot of important information gets stored in a few heads of architects and senior devs. And inevitably, people leave organizations after a while.

A benefit of being a broad generalist is that I can do a lot of things in a lot of languages on a lot of platforms with a lot of frameworks. But I can't remember details anymore. So I have to make things pretty obvious. Functional programming patterns help a lot, because I can usually isolate things into well-named functions which do what they say and nothing more. Then with composition, systems of moderate complexity can be fairly well understood and navigated by readers.

The rest of the post doesn't offer any solutions other than to say, "oh well, that's how it goes". It doesn't have to be a hands-up situation. We as developers must try to balance the "demands" of sprints, product managers, and executives to push stuff out. We should repeatedly explain how technical debt will slow us down in reaching future goals, and how we should refactor and rethink our code regularly. It can be done, and I've successfully changed a few companies. It takes persistence and passion, but eventually you have a whole team onboard with an approach to managing complexity. It is very much worth the struggle.

Re: The collapse of complex software

#212
post #68

Earlier quoted context omitted.

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation. Frameworks approximately do this, but for specific domains: th…

> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That…

This seems like a fantasy considering how big the development teams for these projects are.

Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.

Re: The collapse of complex software

#213
Not complexity guy, competition and innovation. So far the internet is still in its early stage, yes, it is not your “system” but the whole (internet, user, those tidy systems like hbo, Facebook, …). Complexity of the individual system just like any eco-system comes and goes. But what worry is whether the fundamental change (like before and after internet). Just like you observed individuals like us die is not a (perhaps but our individual) system failure. You do not much use basic to do computing or pascal. Even though in 1970s …

Even internet itself has a under threat. That is what we should observe.

Otherwise all is fine. Sone cones some goes, systems moved on. Even mainframe based system moved on.

Re: The collapse of complex software

#214
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down. Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that conven…

The rabbit hole on names goes deep [1]. Lots of other fun rabbit holes. [2]

For how to model physical objects like those cables, go see how Grainger does it as a working practical model.

To really handle names properly, you need more context than the name in the presentation layer that many schemas take their modeling from can obtain. Government health care or similar widely-adopted encoding is sometimes Good Enough. If you want non-lossy exactitude however, then that's a much bigger scope (I'd be investigating a first-pass classifier with contextual hints taken from various geolocations, age, etc., that implies soliciting the name comes after what you normally solicit for input, and refining from there).

Password reset; this is why vendors like Okta exist to abstract authN away for us, and auth0 for authZ. Then there is the rabbit hole of what this abstraction leads to...

[1] https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

[2] https://github.com/kdeldycke/awesome-falsehood

Re: The collapse of complex software

#215
post #174
post #162

Earlier quoted context omitted.

What would be counterexamples to that, i.e. great engineering? FOSS possibly?

seL4. Simple and formally verified. I think it's time to pour VC money on a fully verified stack. Perhaps not everything needs to be proven, integrity for some components can be ensured more informally with e.g. refinement types. Lots of governments and organizations need more robust software.

>I think it's time to pour VC money on a fully verified stack.

While it's true that VCs spend lots of money, you do actually need some plausible-ish excuse for how you could make money on it at some point in the future. It's not like VCs are out here funding public parks or charity hospitals. They fund for-profit businesses.

People did sell operating systems in the 1990s, but I'm not sure if that's a strong market in the 2020s. How are you going to 10x a VC investment in basic software tools?

Re: The collapse of complex software

#216

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

Linux doesn't rewrite? Is writing a completely new I/O scheduler from scratch and replacing the old one not a rewrite? How much does a 5.14 kernel have in common with a 2.6.8 one? If anything, Linux isn't afraid to rewrite the parts that need to be rewritten, and that's one of its strengths. Apache, Vim, and Postgres have been "rewritten." Nginx is an httpd written from scratch that sort of has the same features as A…

With well designed software parts are re-written all the time. When people say rewrite they mean literally rewriting everything from scratch. Stating over.

So if you diff 2.6.8..5.14 course you'll get a massive diff. But they got there by a long series of relatively small changes, with regular releases in between.

Re: The collapse of complex software

#217

Earlier quoted context omitted.

The developers have also gotten too greedy and most of them will job hop every year for the maximum income.

Sounds like if they can job hop every year for maximum income, then the companies they work for are being greedy by underpaying them.

Everyone is greedy, that's how markets work. Every employee is trying to maximize their income and every employer is trying to minimize their expenses. But somehow one of these is seen as "greed" while the other is not.

As a software developer I get paid far above most workers and my job is easier than most other jobs. And yet I could still switch jobs frequently to squeeze out even more money.

Re: The collapse of complex software

#218
If we want a useful, testable theory that "complexity" causes "software collapse", we need to define both independently from each other and find a way to meassure both. Not that I doubt that there is something to be learned here (see for instance the famous Mythical Man-Month). But without a better theory, we are still poking around in the dark. (C64 pun intended.)

Re: The collapse of complex software

#219
post #174

Earlier quoted context omitted.

seL4. Simple and formally verified. I think it's time to pour VC money on a fully verified stack. Perhaps not everything needs to be proven, integrity for some components can be ensured more informally with e.g. refinement types. Lots of governments and organizations need more robust software.

>I think it's time to pour VC money on a fully verified stack. While it's true that VCs spend lots of money, you do actually need some plausible-ish excuse for how you could make money on it at some point in the future. It's not like VCs are out here funding public parks or charity hospitals. They fund for-profit businesses. People did sell operating systems in the 1990s, but I'm not sure if that's a strong market in…

I guess selling secure and verified devices. AdaCore is doing it.

My point is that it can be a very profitable niche, not that VCs fund it pro bono.

Re: The collapse of complex software

#220
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

This is like a philosophers stone of swe or what fred brooks referred to as “silver bullet”, good luck ;)
Post reply on HN