Live data from Hacker News

The collapse of complex software

nolanlawson.com

251–260 of 304 posts

Re: The collapse of complex software

#251

Earlier quoted context omitted.

Linux doesn't rewrite? Is writing a completely new I/O scheduler from scratch and replacing the old one not a rewrite? How much does a 5.14 kernel have in common with a 2.6.8 one? If anything, Linux isn't afraid to rewrite the parts that need to be rewritten, and that's one of its strengths. Apache, Vim, and Postgres have been "rewritten." Nginx is an httpd written from scratch that sort of has the same features as A…

With well designed software parts are re-written all the time. When people say rewrite they mean literally rewriting everything from scratch. Stating over. So if you diff 2.6.8..5.14 course you'll get a massive diff. But they got there by a long series of relatively small changes, with regular releases in between.

Nginx, Neovim and Linux 2.6 were written from scratch to start over from Apache, Vim and Linux 2.4. The rewrite isn't the issue.

The tech debt amassed between the rewrites and then rewriting not to learn from past mistakes but to repeat them, is the issue.

Re: The collapse of complex software

#252
post #28

I highly recommend Rich Hickey's "Simple Made Easy" talk as a way to think about software complexity. We all know that complex systems are hard to work with, but I've found his presentation very useful in understanding the nature of complexity. Understanding complexity helps being able to avoid or minimize it. For example, easy is not the same as simple, and simple changes to one part of a system can introduce great…

I've watched that video at least three times, but I still can't articulate the exact distinction he is making between simple and easy :/

Let's say you have a function that does a lot of different things, and you just call:

  doStuff()
That's easy, right?

But "doStuff()" may be a very complicated function, that's very hard to understand, so it's not simple.

A real-world example might be something like the type conversion that JavaScript and PHP do in comparisons; "just do 0 == '0' and it'll work, easy right?", but turns out those conversion rules aren't simple at all.

Re: The collapse of complex software

#253

Earlier quoted context omitted.

Maybe. I've never worked in an organization where engineers were driving the complexity, nor have I worked in an organization where microservices introduced complexity (I've heard people complain in vague terms about microservices, but I don't relate to their experiences). However, I have worked in an organization that transitioned from monoliths to microservices, and the overall complexity was reduced because (I did…

"Microservices" (ie. SOA) absolutely adds complexity, the question is whether the benefits are worth the overhead. The answer varies according to domain, but is largely a function of team size.

> The answer varies according to domain, but is largely a function of team size.

I agree. If both team size and code base are small, the scalability is not needed and the domain logic isn't too complicated, microservices add an overhead with very limited benefits.

If you have different teams for each service - great. If you the same person is working on multiple services - probably a premature move to a more complicated architecture.

Re: The collapse of complex software

#254
post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

His examples are getting a little dated, but reading this, I can’t help but think of Spolskey’s “Architecture Astronauts” essay: “When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. Tha…

Came to say the same thing, Joel nailed this 21 years ago. Show me the code, and show me what it can do that other stuff can't.

Re: The collapse of complex software

#255
post #68

Earlier quoted context omitted.

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation. Frameworks approximately do this, but for specific domains: th…

> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That…

I thought Linus based Linux off of Minix, but used a monolithic kernel to keep things practical.

Re: The collapse of complex software

#256
post #68

Earlier quoted context omitted.

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation. Frameworks approximately do this, but for specific domains: th…

This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc. All still manageable by humans but at continually higher and higher levels of complexity.

I sometimes wonder if we need to go back to the drawing board or model our problems differently lol.

Re: The collapse of complex software

#257
post #54

Earlier quoted context omitted.

As you implied, I don't think the answer is to make the perfect abstractions that can handle any scenario. But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario. The amount of duplication solving the s…

The problem is "pulling in a tried and tested solution" means pulling in a slice of the whole stack: CSS, Javascript, client/server communication, API endpoints, data-flow logic, and database schema. People are very very bad at this because they believe that software engineering principles apply only to little tiny portions of this stack. If you actually apply your software engineering principles to the whole stack,…

Kinda reminds me of what they do in some areas of the finance world with Kx System's Kdb+. That entire install is tiny tiny and you get a high performance array language for querying your SSD database with either the main language "k" or a syntactical sugar dialect sitting on top called "q" that is more like SQL.

I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.

Re: The collapse of complex software

#258
post #72

Earlier quoted context omitted.

> Names Don't try to force schemas onto schema-less data. Store the "name" as a JSON string/blob representing the various possible attributes (given, middle, family, title, etc) and provide a variety of functions for representing that data. IF you really need to do this at all (for an internal app, you probably don't). > Physical links Include an Hardware Asset FK in your Link M2M table. Model each binary link explic…

> Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly. Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example). So if you m…

Can't you just have 2 tables? One for 2 end and one for 3 ends?

Re: The collapse of complex software

#259

I appreciated this essay. Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job." While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard .…

There is the well-worn Pascal quote about having no time to write a shorter letter[1]. There is also, I think, an important point hidden in the “domain knowledge” part: the freedom to change the spec ; the effusive texts on Charles Moore’s approach to Forth programming[2,3], however much scepticism they deserve otherwise, are very direct about that freedom being essential. But that means simplicity is a cross-cutting…

> Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.

> I don’t see a good solution to this.

I think this is probably true.

Some people take that to basically give up, and say, okay, our software is always going to be a mess, but look at all it's done as a mess, so be it.

Otherwise... the commonly understood solution seems to be composing the thing of independent units (very very very decoupled) each of which can fit into the mind of a couple of people at most. And then the whole, built of these independent units considered as black boxes, can perhaps also fit into the mind of a couple of people at most. I know you said "the complete edifice", which is not the same thing, but this is what we've got: and indeed is "abstraction", basically the entire basis of computer science and what makes possible anything we do. (Even the simplest program of 20 years ago -- can probably only fit into the mind of one or two people if you exclude the hardware all the way down, which you can because it is well abstracted and decoupled; it wasn't always, 50 years ago).

And we can come up with all the challenges and barriers of that -- of course! If it were easy, we'd have simple software. :)

But that's the task, I guess. But a lot of software isn't even built realizing that's the task -- to have one or two people "driving the bus" who have a mental model of the thing they're building, at any level of abstraction (I feel like Fred Brooks wrote about this), meaning that continuity of experience matters and domain knowledge matters, treating developers as commodities to be shifted around continually has a cost to complexity too, as does continually switching technologies and platforms so you can't build up a solid repertoire of encapsulated abstracted pieces to compose.

Re: The collapse of complex software

#260
One great tool for fighting complexity is to Always Consider Option Zero: whenever comparing possible solutions to a problem, consider the option (Option Zero) where you simply don’t solve the problem. It’s amazing how much it changes the conversation when you stack up the pros and cons of the status quo, against the pros and cons of each solution. Quite often you realize the best path is much simpler than you thought.

It’s not a panacea, but it really does help slow the growth of your complexity.

Post reply on HN