Quick mental exercise on this.
If someone posed this question to you in an interview and you used these principles, would you get the job?
Probably not.
Practices of Reliable Software Design
21–30 of 58 posts
Re: Practices of Reliable Software Design
#22There is a bunch of good advice here, but it's missed the most useful principal in my experience, probably because the motivating example is too small in scope: The way to build reliable software systems is to have multiple independent paths to success. This is the Erlang "let it crash" strategy restated, but I've also found it embodied in things like the architecture of Google Search, Tandem Computer, Ethereum, RAID…
Interestingly this is exactly how I've come to define truth/correctness: https://alexpetralia.com/2023/01/25/how-do-we-know-if-data-i...
Re: Practices of Reliable Software Design
#23There is a bunch of good advice here, but it's missed the most useful principal in my experience, probably because the motivating example is too small in scope: The way to build reliable software systems is to have multiple independent paths to success. This is the Erlang "let it crash" strategy restated, but I've also found it embodied in things like the architecture of Google Search, Tandem Computer, Ethereum, RAID…
I disagree somewhat, influenced by the teachings of Nancy Leveson. In the 1930's, yes, component redundancy was the way to reliability. This worked at the time because components were flaky and technical systems were simple aggregations of components. Today, components themselves are more reliable, but even when they are not, redundancy adds only a little reliability because there's a new, large, source of failure: i…
This is why Erlang's OTP focuses on supervisor trees. At each level of the component hierarchy, you have redundancy. Subcomponents themselves may have interactive complexity, but a failure or misspecification in any of the interactions making up that subcomponent simply makes that subcomponent fail. This failure is handled at a higher level by doing something simpler.
And "do something simpler" is actually a core part of this strategy. You're right that "today's systems are so complicated that many failures stem from insufficient, misunderstood, or ambiguous specifications". In most cases, yesterday's system worked just fine, you just can't sell it as a competitive advantage. So build simple, well-understood subsystems as fallbacks to the complex bleeding-edge systems, or even just take the software that's been working for a decade.
Re: Practices of Reliable Software Design
#24Earlier quoted context omitted.
the simple basic reality of statistics, a binomial distribution. 5 independent systems with 90% chance of success is mathematically as reliable as one that is 99.999%. 100x 90% systems would get you to 100 "9s" of reliability aka 99.99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999%
Actually making 5 completely independent systems would be exceptionally hard. No shared code or team members, no shared hardware... For example, what 5 computing platforms would you use? x86, ARM, RISC-V and...? Math rarely applies so easily to real life. Talking about "independent" systems is cheap. If at all possible. How would you transport yourself to work using two independent systems?
It's relatively simple at the organizational level, just expensive (but linearly expensive, while often increasing subcomponent reliability is exponentially expensive!). Just give the same problem statement to two independent teams with two different managers, have a clear output format and success criteria, and let them make all their technical decisions independently.
Your example of "how do you transport yourself to work using two independent systems" is actually very apropos, because I and many other commuters do exactly that. If the highway is backed up, I bypass it with local roads. If everything is gridlock, I take public transportation. If public transportation isn't functioning (and it generally takes a natural disaster to knock out all the roads and public transportation, but natural disasters have happened), I work from home and telecommute. Each of these subsystems is less favored than the alternative, but it'll get me to work.
Re: Practices of Reliable Software Design
#25There is a bunch of good advice here, but it's missed the most useful principal in my experience, probably because the motivating example is too small in scope: The way to build reliable software systems is to have multiple independent paths to success. This is the Erlang "let it crash" strategy restated, but I've also found it embodied in things like the architecture of Google Search, Tandem Computer, Ethereum, RAID…
In the limit, there is a hard tradeoff between efficiency and reliability. Failovers, redundancies, and backups are all important for building systems that are resilient in the face of problems, for reasons you've pointed out. However, failovers, redundancies and backups are inefficient. Solving a problem with 1 thing is always going to be more efficient that solving the same problem with 10 things. It's interesting…
This is a very important point, and often misunderstood on both a business & societal level. Reliability has a cost. If you optimize all redundancy out of a system, you find that the system becomes brittle, unreliable, and prone to failure. Companies like 3M and Boeing have found that in the pursuit of higher profits, they've lost their focus on quality and suffer the resulting loss of trust and brand damage. The developed world discovered that with COVID, our just--in-time efficiency meant that any hiccup anywhere in the supply chain meant mass shortages of goods.
Re: Practices of Reliable Software Design
#26My first thought upon seeing the prompt: If you would build an in-memory cache, how would you do it? It should have good performance and be able to hold many entries. Reads are more common than writes. I know how I would do it already, but I’m curious about your approach. Was to add this requirement since it comes up so often: Let's assume that keys accessed follow a power law, so some keys get accessed very frequent…
I've been doing this so long that my first thought was "use redis."
Why?
* it works
* it's available now
* it scales
* it's capable of HA
* it has bindings for every language you probably want to use
Why bother writing your own cache, unless it's for an exercise? Cache management is complicated and error prone. Unless the roundtrip kills you just use redis (or memcached).
Re: Practices of Reliable Software Design
#27My first thought upon seeing the prompt: If you would build an in-memory cache, how would you do it? It should have good performance and be able to hold many entries. Reads are more common than writes. I know how I would do it already, but I’m curious about your approach. Was to add this requirement since it comes up so often: Let's assume that keys accessed follow a power law, so some keys get accessed very frequent…
Re: Practices of Reliable Software Design
#28My first thought upon seeing the prompt: If you would build an in-memory cache, how would you do it? It should have good performance and be able to hold many entries. Reads are more common than writes. I know how I would do it already, but I’m curious about your approach. Was to add this requirement since it comes up so often: Let's assume that keys accessed follow a power law, so some keys get accessed very frequent…
I think splay trees would be good for this: https://en.m.wikipedia.org/wiki/Splay_tree
Re: Practices of Reliable Software Design
#29> It is much easier to add features to reliable software, than it is to add reliability to featureful software. Not sure about this tbh. In a lot of cases yeah maybe. But when you are dealing with complicated business logic where there is a lot of bells and whistles required, building a simple reliable version can lead you into a naive implementation that might be reliable but very hard to extend, while making an uns…
How are you defining simple here?
Simple in my mind has abstractions where they are needed which should naturally lead to easy to extend code.
Re: Practices of Reliable Software Design
#30Great points.
But why do we invest so much complexity into outputting html/js/css.