This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coprocessor" assist in seeing the solutions. I believe there are well-known strategies (taken from the real world) for managing hierarchies of complexity that ease the cognitive load of exploring any problem. We just need to get these approaches into a programming "langugage"!
The collapse of complex software
11–20 of 304 posts
Re: The collapse of complex software
#12Our industry's "solution" to complexity is somewhat unique: We sweep it under the carpet. The reason that actually works is that the marginal cost to reproduce an existing pile of complexity is zero.
Once we've got a sufficiently complex pile that actually has promise, everybody duplicates the pile. And then puts an additional layer on top. These many pile+layer combos then co-evolve (and co-erode ;)
At some point, something so useful has evolved that we start copying that pile... and put new layers on top. Rinse. Repeat.
The complexity rarely collapses because those new layers can add support beams, if necessary. Worst case, a layer gets ablated, and we try again from one step below. We don't ever go back to the beginning.
(This also explains, to me, why Enterprise software is what it is - the complexity piles can rarely be shared or reused, and so there's no co-evolution, little chance to settle on the best pile)
And, to be fair, it's similar in other disciplines, but maybe a bit slower. Mechanical engineering has individual components, larger components built from those, and so on. It just is more expensive to copy the piles^W components we know to work.
Total collapse is exceedingly rare.
(That even holds for human societies - they, too, often "just" collapse down to simpler societal expressions. And we do co-evolve larger and larger forms of society. It's just that the erode-and-try-again process is uniquely painful for the humans involved)
Re: The collapse of complex software
#13This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…
It's a solved problem, if you are solving the right one. Most companies out there are solving problems they don't have. The FAANG disciples descended upon the world to preach the gospel of Complexity, and everyone bought it - because everyone wants to be like Google.
What, you think "microservices" is a new invention? We called it distributed systems, and we really knew not to go there unless we wanted to decimate our productivity and sleep.
Re: The collapse of complex software
#14I like to believe that serverless technologies and cloud services reduce complexity for the org, but obviously that's at the cost of offloading that complexity to the cloud providers (and welding an org's software to that provider for years, if not decades). So different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.
They absolutely do reduce complexity! For example, think about building, testing, and deployments.
In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose job it was to solely maintain this infrastructure along with database tuning, build fleet monitoring, the list of menial tasks just goes on and on and on.
Today, you have a yaml file in your .github/workflows directory.
Now I agree that there are "different" requirements. Understanding the complexity of your workflows etc is no small feat-- but you're replacing such a huge amount of what used to be extremely expensive and brittle architecture with, basically, a text file or two. That's a huge cost savings.
Re: The collapse of complex software
#15This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…
Good luck but your motivations will probably not line up with the masters who dictate things that become complexity.
For example, I look at our software running on .net MVC and think as an Engineer, it simple and knowable but the Front-End Team are less worried about simplicity and more on flashy front-end stuff since that is their job. We end up bolting on a Front-end JS framework and complexity immediately ramps up by like 300%.
Are they wrong for wanting a better and more flexible front-end? Not necessarily, I mean all the other companies have cool stuff and if we don't maybe our company dies.
Ditto for lots of other examples...
Re: The collapse of complex software
#16This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…
this reminds me of the Head First series of books. I can still recall specific pages from the Head First Statistics book not because of the content but because every topic (and probably page) had a creative visualization.
Re: The collapse of complex software
#17Note that Google's "Frequent rewrites" practice -- see section 2.11 of https://arxiv.org/pdf/1702.01715.pdf -- substantially reduces the severity of this problem (though it requires a good higher-level architecture).
Re: The collapse of complex software
#18> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads
Where does this yutz work? Because he doesn't speak for me. Generally, I write a complex program because I don't have time to write a simpler one; and when I have to deal with that complexity later I'm like arrrggh. Rube Goldberg machines are fun to watch, but no one wants to maintain, change, or build on one.
Re: The collapse of complex software
#19I like to believe that serverless technologies and cloud services reduce complexity for the org, but obviously that's at the cost of offloading that complexity to the cloud providers (and welding an org's software to that provider for years, if not decades). So different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.
They absolutely do reduce complexity! For example, think about building, testing, and deployments. In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose…
Are you counting Chef/Ansible/Puppet/etc as things that came during the cloud era or that those things were not used to solve the problems you raised before the cloud era?
Re: The collapse of complex software
#20This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…
Me too, and I definitely agree that it is time to move past thinking in "languages" to a more holistic view of programming systems. The fact that as of now, only ~1% of people can access programming and wield the magic of the computer is a tragedy akin to the comparable levels of literacy in the medieval era. At the present, there seems to be a minor renaissance of thought along these lines stemming back to the pioneers of the field (McCarthy, Engelbart, Kay, etc). Everyone has a different take as to what the answer is, but I think "let 1000 flowers bloom" is the best way to find a solution to a complex and largely unknown problem such as software. Personally, I'm afraid that if we fail, the means of programming will become gated behind AIs and society will continue in its trance-like state of passivity instead of creative expression.