The collapse of complex software • Hacker News Board

Re: The collapse of complex software

#121

Post by astrange » Thu, Jun 09, 2022, 7:49 PM UTC

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

This is essentially Alan Kay/VPRI's old research program. I'm not sure why, but when people get into this they always think dataflow programming will solve everything - we already have Excel and it does do a lot, but it doesn't do everything.

Re: The collapse of complex software

#122

Post by notatpeace » Thu, Jun 09, 2022, 7:50 PM UTC

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

May be you are looking at a much fine grained composition/decomposition tier to arrive at breaking down "all" software engineering towards simplicity, but anything real world usecase would end up being an aggregate so big, your composition/decomposition tier itself would add to complexity rather than reduce it.

BPMN and all of the UI pallet, drag drop coding applications try to do this very thing, this is super successful at smaller scales but breaks the first real world application.

Re: The collapse of complex software

#123

Post by DoneWithAllThat » Thu, Jun 09, 2022, 7:51 PM UTC

turminal wrote: post #118

jacobsenscott wrote: post #107

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

> Things like the linux kernel, all the GNU utils, emacs, vim, apache, nginx, postgres. The list goes on. We know how to write software that lasts for decades, continuously improves, and doesn't need constant re-writes. At least half of the items on that list are still here purely on momentum and reputation from the time there were no alternatives, not because they were really that marvelous. I use and even enjoy som…

I would disagree with that sentiment for every single item on that list. All those tools are amazing and have improved steadily over decades and I would absolutely put them in the “great engineering” category.

Re: The collapse of complex software

#124

Post by astrange » Thu, Jun 09, 2022, 7:51 PM UTC

nightski wrote: post #61

notriddle wrote: post #58

Earlier quoted context omitted.

> Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. The type signature of GPT-3 is `string -> string`

Well I'd argue it's more Model => String => String if you are talking about using a fully pre-trained model. Even then you could probably expand on the type signature of the model to make it more useful. But if you look at like co-pilot for example, if that was given the ability to have type signatures serve as input you might get a lot more powerful results than what it does with raw text (which is very impressive).…

Type signatures (or ASTs) are structured data, and Copilot mainly works because GPT accepts any text and doesn't have to deal with your logical fully recursive concepts like ASTs.

I feel like it wouldn't work if you could actually mathematically constrain the outputs to be syntactically correct either. That's probably one of those Gödel things.

Re: The collapse of complex software

#125

Post by phendrenad2 » Thu, Jun 09, 2022, 7:52 PM UTC

sethetter wrote: post #111

jacobsenscott wrote: post #107

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

I'm curious how big of a factor a consistent core team is to having long-term sustainable and stable software. When companies have people rotating on projects constantly and the original vision and design principles of the product are lost or changed, I can only imagine that contributes to the problem.

I think that's a big factor. This is what RSUs and bonuses are supposed to solve: You entice employees to stay rather than job hop. However, companies have gotten too greedy and RSUs amount to a pittance, a vestigial perfunctory play act mimicking a bygone era.

Re: The collapse of complex software

#126

Post by nescioquid » Thu, Jun 09, 2022, 7:53 PM UTC

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

I attended a Common Lisp convention at MIT around the year they did away with the SICP course and replaced it with a course using Python to program robots. It came up in the Q&A of one of the talks and the question got kicked to Sussman who was in the audience and gave an explanation.

He said that it used to possible, practically speaking, to treat software like discrete components so that you can reason about how to combine the units and predict the result, etc. By contrast, much programming today is bodging together a bunch of libraries whose behaviors are often poorly defined or otherwise opaque so that one often enough has to treat them like black boxes against which you have to apply scientific reasoning to understand how to use and integrate it. Hence Python and robots.

Presumably the new course teaches skills around managing this complexity, but the shift in model advocated -- from the metaphor of well-behaved discrete components in composition to opaque units whose inputs and outputs (possibly depending on hidden state) must be discovered was interesting to me, and ultimately adds to the accidental complexity of doing software now.

Do you think this sort of problem may have a distinct visual representation that would fundamentally reduce the complexity of library-dominated development? Or are you imagining a whole programming ecosystem built around your idea, so that the fundamental problem Sussmen described is somehow obviated? Or are you really trying to address essential, not accidental complexity?

Re: The collapse of complex software

#127

Post by dale_glass » Thu, Jun 09, 2022, 7:54 PM UTC

jacobsenscott wrote: post #107

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

Many of those did need something approaching a rewrite and even got one of sorts, but just were big enough that there was plenty that was unaffected by whatever big change was needed.

The Linux kernel for instance gets big changes. The BKL took a very long time to fully remove.

The GNU utils needing a rewrite would be weird since most of them implement a pre-existing standard, so the need for a huge rewrite is very unlikely.

Re: The collapse of complex software

#128

Post by bezospen15 » Thu, Jun 09, 2022, 7:55 PM UTC

jacobsenscott wrote: post #107

This "software grows too complex" is certainly a culture issue, and not a software problem. It mostly comes from businesses throwing a bunch of under-skilled labor at the problem, and demanding more and more too quickly. Of course it will outgrow the ability of that team to work with, reason about, keep clean, and extend. They will get frustrated and move on, and a new group comes in, re-writes it from scratch, and s…

Odd

I thought it was caused by every little piece of software being tied to giant product teams full of MBA majors that want to add 50 features a week

Re: The collapse of complex software

#129

Post by eranation » Thu, Jun 09, 2022, 7:55 PM UTC

> Simplicity of design sounds great in theory, but it might not win you many plaudits from your peers. A complex design means more teams to manage more parts of the system, more for the engineers to do, more meetings and planning sessions, maybe some more patents to file. A simple design might make it seem like you’re not really doing your job. “That’s it? We’re done? We can clock out?” And when promotion season comes around, it might be easier to make a case for yourself with a dazzling new design than a boring, well-understood solution

I couldn’t disagree more. The vast majority of companies I worked with recognized and rewarded simpler designs, the vast majority of software engineers I worked with would rather delete code than write code, rather throw away a complex design to a simpler one, and with some NIH here and there, YAGNI and KISS seems to be triumphant and what would get you promoted, not adding more arrows and boxes.

Re: The collapse of complex software

#130

Post by yellowapple » Thu, Jun 09, 2022, 7:58 PM UTC

perlgeek wrote: post #37

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down. Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that conven…

> What's the "right" way to store person names?

Store both a unified name field and separated given name / surname fields and let the user manage both. Yeah, this risks going out of sync, but that's the user's problem, not yours. Yeah, three fields are technically more complex than one or two, but it produces less complexity down the line.