Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The collapse of complex software

nolanlawson.com

Post reply on HN
Print view View original

41–50 of 304 posts

Re: The collapse of complex software

#41

Post by mark242 »

mark242 wrote: post #14
parsley27 wrote: post #2

I like to believe that serverless technologies and cloud services reduce complexity for the org, but obviously that's at the cost of offloading that complexity to the cloud providers (and welding an org's software to that provider for years, if not decades). So different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.

They absolutely do reduce complexity! For example, think about building, testing, and deployments. In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose…

Another quick example - serving up product image assets to customers. Let's say you want to ship 1TB of images to customers per month.

In pre-cloud times, you would spend untold piles of money spinning up racks of storage arrays, switches, firewalls, leasing out gigantic pipes for bandwidth, and again all of the requisite people in order to get that running. You'd rent space in multiple distributed global datacenters, so again you'd have an unreal amount of bash-script-file-sync services so that when someone uploads a new image it gets replicated all over the globe. Millions and millions of dollars. You'd probably have to write a custom resizing service with ImageMagick, hooked into your frontend, so that you were serving customers the correct size and not blowing through your bandwidth allocation. Just incredibly complex.

Today, you click a button in your CDN provider's console; most of that functionality from above just comes for free.

Again you have to do a little bit of munging your front-end to take into consideration the vagaries of your CDN provider, but overall it's such a huge savings of mental energy and time. Put a circa-1999 systems architect in front of the Cloudflare console from today and they wouldn't believe it was real.

Re: The collapse of complex software

#42

Post by nightski »

perlgeek wrote: post #37
daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down. Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that conven…

As you implied, I don't think the answer is to make the perfect abstractions that can handle any scenario.

But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario.

The amount of duplication solving the same problems is insane. But this is not an easy problem to solve and I don't intend to trivialize it.

I think we need to come up with better tools. code sharing through libraries/repositories (like npm) is great, but it can't be the final solution.

Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. You could import modules, functions, data structures, anything. But that vision is still a long ways off.

Re: The collapse of complex software

#43

Post by chasd00 »

bitwize wrote: post #18

> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads Where does this yutz work? Because he doesn't speak for me. Generally, I write a complex program because I don't have time to write a simpl…

didn't Mark Twain say something like "i'm sorry this letter is so long, i didn't have time to write a shorter one". I feel like that with some of my code at times.

Re: The collapse of complex software

#44

Post by michaelbuckbee »

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

What you're describing is one approach, the other is that you pay a bunch of SaaS/PaaS/IaaS companies to handle a lot of the workload for you so you and a small team can just focus on the business logic of your particular niche.

Re: The collapse of complex software

#45

Post by chasd00 »

eldude wrote: post #21

Complexity in software is a lot like complexity in cuisine; most chefs want to stretch their abilities and create something memorable, transcendent even, but at the end of the day unless they’re cooking for themselves or friends and family there’s a business to be run. So complexity has a diminishing return for economical reasons.

i kind of like that analogy. And many times, a ham sandwich and a beer really hits the spot.

Re: The collapse of complex software

#46

Post by mark242 »

beardedetim wrote: post #19
mark242 wrote: post #14

Earlier quoted context omitted.

They absolutely do reduce complexity! For example, think about building, testing, and deployments. In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose…

Are you counting Chef/Ansible/Puppet/etc as things that came during the cloud era or that those things were not used to solve the problems you raised before the cloud era?

Having used Puppet, the promise of Puppet was very different than the realities of Puppet and imo more accurately reflects the "different complexities" point. The point of those provision-bare-metal-on-the-fly frameworks just evaporated when, for example, you could launch a container on Fargate with, again, a single text file, or similar on Heroku.

Re: The collapse of complex software

#47

Post by greatpostman »

_xnmw wrote: post #6

I'm waiting for the day that AWS collapses under its own weight. Heard horror stories from backend engineers working on the infrastructure/dashboard code there.

I think there’s basically a zero percent chance of this happening. Aws may internally be full of technical debt, but for ten years now Amazon engineers have been automating the reliability issues of the platform. The capabilities are expanding, but the core components are some of the most rock solid engineering on the planet.

Re: The collapse of complex software

#48

Post by mathgladiator »

daenz wrote: post #11

This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coproces…

Welcome to the club!

The fundamental problem of simplifying software is humans. Just consider date formats, time zones, and tax codes. Humans love to make complicated things.

My philosophy on this has been to take the complex human stuff and stick it in a black box. A professional feather in cap with this approach is called BladeRunner ( https://dl.acm.org/doi/10.1145/3477132.3483572 ) which radically simplified the distributed system aspect by putting all the gnarly glue and business logic in a V8 VM (JavaScript).

My next thing is a focus on board games where I have invented a programming language and have started to evolve a platform. It's called Adama ( https://www.adama-platform.com/ ), and I think it is pretty cool. The interesting thing is that the complexity of board games is exceptional.

I have a few clues to share. The first is that reactivity, which is found in excel, is a key to simplifying software as this makes the glue more automatic.

Another clue is figuring out bidirectional communication which relates to reactivity as a two-way street. However, this is primarily hard because we don't have great things off the shelf to deal with this beyond TCP. For more of a deep dive, check out https://www.adama-platform.com/2021/12/22/woe.html which talks about WebSocket.

My final clue is that you can't run away from state. So many people offload state because state is hard, and you have to contend with it. I'm building yet another database.

Re: The collapse of complex software

#49

Post by nonrandomstring »

Very nicely written. A lot of gems in this short essay. What stood out for me was:

> It takes a lot of discipline to resist complexity

That can be taken in many ways.

Self-discipline, to resist convenience, the omniscient and omnipotent desire to control everything and always have the right answer immediately. That takes discipline to let go of. Also to discipline others, to say no, push back, become the master and set firm boundaries against crushing 'demands' for complexity. And intellectual discipline, to see clearly, elegantly, eliminating what is unnecessary. Over-complexity is a failure of the ordering principle of an organism, a step on the road to chaos and entropy.

Re: The collapse of complex software

#50

Post by undershirt »

chrchang523 wrote: post #17

Note that Google's "Frequent rewrites" practice -- see section 2.11 of https://arxiv.org/pdf/1702.01715.pdf -- substantially reduces the severity of this problem (though it requires a good higher-level architecture).

> 2.11. Frequent rewrites

> Most software at Google gets rewritten every few years.

> This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. However, it also has some crucial benefits that are key to Google’s agility and long-term success. In a period of a few years, it is typical for the requirements for a product to change significantly, as the software environment and other technology around it change, and as changes in technology or in the marketplace affect user needs, desires, and expectations. Software that is a few years old was designed around an older set of requirements and is typically not designed in a way that is optimal for current requirements. Furthermore, it has typically accumulated a lot of complexity. Rewriting code cuts away all the unnecessary accumulated complexity that was addressing requirements which are no longer so important. In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”. Frequent rewrites also encourage mobility of engineers between different projects which helps to encourage cross-pollination of ideas. Frequent rewrites also help to ensure that code is written using modern technology and methodology.

Post reply on HN