Live data from Hacker News

How to prevent software bug from killing lives like Boeing's or Tesla's?

news.ycombinator.com

11–20 of 20 posts

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#11
Follow engineering best practices. One thing you should never do is to write a new system, requiring it to perfectly emulate an old one. It can never work and there will always be unexpected deviations in system behavior. I.e. in Boeing's case it isn't so much about any particular flaws (faulty AoA sensor etc), it is about the whole idea of having a completely new aircraft design appearing to the pilot as if it was an old one. It is similar to replacing an Active Directory deployment with OpenLDAP and betting on users not noticing.

Another engineering best practice is to keep proper logs. The Toyota recalls from 2009 to 2011 were likely caused by some bugs but they weren't able to find the root cause. Ostensibly because not enough data were being logged.

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#12
Process process process.

Start with researching formal "quality management systems." At the very least that'll introduce you to using FMEAs, external standards, rigorous design reviews and testing, quality gates between development and production (if they're not a PITA, they're not working) and traceability for everything in production.

There are off-the-shelf learning materials for all of it.

If you do go down that road hire someone with QMS experience to design your process and handhold your team through the transition. Otherwise you're likely to over-complicate it for a less effective result.

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#13
post #9

Overall, I think designing a system that prevents people from being harmed is very hard, and has to be a design level concept that is in everyone's minds as they start, and every day through development. Even your example "the worst can happen when our software has problem is someone won't get a meal on time or a meal is wasted." isn't really true. What if you ordered fish, or oysters, and they were left out too long…

Thank you for very interesting thoughts. Learned something new.

You were right with the counter-exaple that we have redundancy in place that is 'hardware' - mealbox stickers, production qa.

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#14
post #11

Follow engineering best practices. One thing you should never do is to write a new system, requiring it to perfectly emulate an old one. It can never work and there will always be unexpected deviations in system behavior. I.e. in Boeing's case it isn't so much about any particular flaws (faulty AoA sensor etc), it is about the whole idea of having a completely new aircraft design appearing to the pilot as if it was a…

Completely agree. Logs are so important when bugs happen.

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#16
post #11

Follow engineering best practices. One thing you should never do is to write a new system, requiring it to perfectly emulate an old one. It can never work and there will always be unexpected deviations in system behavior. I.e. in Boeing's case it isn't so much about any particular flaws (faulty AoA sensor etc), it is about the whole idea of having a completely new aircraft design appearing to the pilot as if it was a…

Interesting point on keeping good logs. Do you know of any best practices? I wonder if there are any reference texts that articulate rules on writing logs for coders/software developers in training?

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#17
For minimizing disaster, the standard grad text for this type of thing is Requirements Engineering: From System Goals to UML Models to Software Specifications by Axel van Lamsweerde. There you learn about creating models and risk analysis, fault tolerance modeling[1], privacy requirements etc., all established methods based on engineering foundations but applied to software modeling and development.

If you mean software development for mission critical things that control movement like aircraft, drones, factory robots etc., these engineers I would assume use verified compilers/toolchains like CompCert project to implement the models they have already formally analyzed http://symbolaris.com/course/fcps17.html but I've never done mission critical work, just dabbled in it to apply methods there to non-critical software.

[1] https://arxiv.org/pdf/1611.02273.pdf - Application-layer Fault-Tolerance Protocols

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#19
A simple approach would be to not automate things that don't need automating. I think as developers we get a little too far ahead of ourselves.

The failure on the MAX was related to computers moving controlled surfaces (flaps, rudder, etc.) on the plane automatically based on sensor readings. Questions should be asked about what problem that was solving, and was that a requirement (aka complaint) from pilots? Why was that feature necessary?

Re: How to prevent software bug from killing lives like Boeing's or Tesla's?

#20

A simple approach would be to not automate things that don't need automating. I think as developers we get a little too far ahead of ourselves. The failure on the MAX was related to computers moving controlled surfaces (flaps, rudder, etc.) on the plane automatically based on sensor readings. Questions should be asked about what problem that was solving, and was that a requirement (aka complaint) from pilots? Why was…

I read that it was due to the new design of their engine which is more likely to lead to a 'stall' situation, hence, they put in 'anti-stall' system
Post reply on HN