Ask HN: High availability control system software architectures

#1

Post by zeus_hammer » Fri, Dec 03, 2021, 9:20 PM UTC

Hi HN. I'm interested in resources describing the architecture and implementation of critical control system software. I'd like to understand more specifically about how these systems can be or are designed such that loading, patching, and deploying software in these environments can be done with zero downtime.

Are there any books, code, or other resources you would recommend?

Re: Ask HN: High availability control system software architectures

#2

Post by yuppie_scum » Sat, Dec 04, 2021, 12:23 AM UTC

Some topics for you to research:

12 Factor Applications

Site Reliability Engineering

Chaos Engineering

Kubernetes

Re: Ask HN: High availability control system software architectures

#3

Post by GianFabien » Sat, Dec 04, 2021, 1:37 AM UTC

Telephony central office switches are an example of the sort of systems you are asking after. A typical switch is designed to run for 30 years with less than 30 minutes downtime in all that time. Ericsson AXE and Western Electric are two such systems I have worked with. More recent example would be the Erlang language and the related OTP environment.

Re: Ask HN: High availability control system software architectures

#4

Post by chermi » Sat, Dec 04, 2021, 2:20 AM UTC

Can you specify exactly what you mean by "control system" here? Are you talking about software that actually sends signals to hardware that eventually makes to physical equipment that does something?

I'm trying to understand the actual environment. When you say "deployment", what is changed, where does it start, and how far does it propagate?

For example, would one option for zero downtime be to have replicated (2 or more) "control systems" beyond some "layer" (sorry, it's hard to be precise without knowing more) and enforcing synchronicity between those while having only actually controlling at any time. Then, when you are patching or updating, you freeze on one, update the other, then switch to the other? Not advocating a solution, just trying to understand the situation by throwing out an example to talk around.

I'm not an expert in this at all, but if what I'm talking about above is even close to being on track, I'd recommend this book for starters: https://www.amazon.com/Introduction-Embedded-Systems-Cyber-P...

Re: Ask HN: High availability control system software architectures

#5

Post by karmakaze » Sat, Dec 04, 2021, 2:24 AM UTC

Loading, patching, and deploying without downtime is not very complicated on the surface of it. Basically almost all cloud product/service providers do this with fault-tolerant network design, routing/load-balancing, distributed/fault-tolerant datastores, blue-green continuous integration/delivery (CI/CD) pipelines.

The hard part is being very strict to ensure that every change is safe and/or be able to rapidly/automatically restore a working state to stay within a very low error budget. Each '9' in 99.9.. of uptime is order(s) of magnitude harder.