The Three Levels of Software Safety
bellmar.medium.com
The Three Levels of Software Safety
1–10 of 13 posts
Re: The Three Levels of Software Safety
#2> Safety is ultimately about preventing a system from reaching dangerous states.
This is a fallacy. Whether a state is dangerous or not often depends on the environment outside the system and usually cannot be totally foreseen by the engineer.
I happily damage my engine if I can drive out of the dessert.
Depending on case I’m happy if files can be undeleted - or not.
But of course I appreciate a warning why the state could be dangerous.
Re: The Three Levels of Software Safety
#3For an article which aspires to clarify safety I would expect a good definition of the term. Is it "preventing a system from reaching dangerous states"?
In my opinion, you should start with defining what harm to humans you think about. Harm due to privacy leaks? Financial loss due to miscalculations? Locked out by your home security system? Electrocuted by your home security system?
I'm working in driver assistance, safety-critical software in cars. Here is a very high level description how we treat safety:
We start with a high level feature, e.g. automated emergency breaking (AEB). Now someone determines safety goals like "don't break unintentionally" and "don't break too hard". Violating those goals would probably mean the driver loses control of the car and thus harms humans. It needs to be a more specific, of course. Breaking too hard is "decelerate no more than 60km/h in 1.2 seconds" if I remember it correctly. Note that all usual features always use the human driver as fallback mechanism. That results in very interesting questions for autonomous driving.
From those high level requirements it traces down to various parts (hardware, software, electric, mechanics, whatever). There might be a requirement like "this device (ECU), never send signal AebBreakRequest=1 for longer than 1.2seconds".
Then the tedious work comes. You think through all the parts and safety goals and wonder: If something goes wrong here, would it violate the safety goal? If you don't find a good argument, then you need to build some mechanism to prevent it from going wrong. Examples: What if a bit flips in RAM? Use ECC RAM then the probability is low enough. What if there is noise on the CAN vehicle bus? We do check sums. How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence. Those question and answer games can become quite elaborate.
Note that the general pattern here is "if there is a fault, will a safety goal be violated." These days Safety Of The Intended Function (SOTIF) comes with a new pattern "if everything is working as intended, is it safe?" This is mostly triggered from autonomous driving and deep learning approaches. Absent any faults like bit flips, how can we be sure that it won't kill pedestrians dressed up as chickens?
Re: The Three Levels of Software Safety
#4> Safety is ultimately about preventing a system from reaching dangerous states. This is a fallacy. Whether a state is dangerous or not often depends on the environment outside the system and usually cannot be totally foreseen by the engineer. I happily damage my engine if I can drive out of the dessert. Depending on case I’m happy if files can be undeleted - or not. But of course I appreciate a warning why the state…
Identifying what a dangerous state is is usually left to the person writing the requirements. Often that's a completely different department from the engineers who implement them.
Re: The Three Levels of Software Safety
#5> Safety is ultimately about preventing a system from reaching dangerous states. This is a fallacy. Whether a state is dangerous or not often depends on the environment outside the system and usually cannot be totally foreseen by the engineer. I happily damage my engine if I can drive out of the dessert. Depending on case I’m happy if files can be undeleted - or not. But of course I appreciate a warning why the state…
Identifying what a dangerous state is is usually left to the person writing the requirements. Often that's a completely different department from the engineers who implement them.
You are lucky to have someone write requirements who weights in these aspects. I have seen too many specs (this mostly happens in business software) where failures are knowingly or unknowingly ignored... because schedule.
I think failures should be designed for (or at least explicitly defined) even if the requirements do not include them.
Re: The Three Levels of Software Safety
#6Earlier quoted context omitted.
Identifying what a dangerous state is is usually left to the person writing the requirements. Often that's a completely different department from the engineers who implement them.
You are lucky to have someone write requirements who weights in these aspects. I have seen too many specs (this mostly happens in business software) where failures are knowingly or unknowingly ignored... because schedule. I think failures should be designed for (or at least explicitly defined) even if the requirements do not include them.
I think that's a lot more common in software where safety matters. Often such software is developed according to some regulations like ISO26262.
Re: The Three Levels of Software Safety
#7Earlier quoted context omitted.
You are lucky to have someone write requirements who weights in these aspects. I have seen too many specs (this mostly happens in business software) where failures are knowingly or unknowingly ignored... because schedule. I think failures should be designed for (or at least explicitly defined) even if the requirements do not include them.
I think that's a lot more common in software where safety matters. Often such software is developed according to some regulations like ISO26262.
Yes, I have to constantly remind other front-end developers that we don’t need to be pixel-perfect faithful to Zeplin or the story. It is more of a guide and you can always bring up any concerns you have or if you think something should be done differently and we can talk about it.
Re: The Three Levels of Software Safety
#8For an article which aspires to clarify safety I would expect a good definition of the term. Is it "preventing a system from reaching dangerous states"? In my opinion, you should start with defining what harm to humans you think about. Harm due to privacy leaks? Financial loss due to miscalculations? Locked out by your home security system? Electrocuted by your home security system? I'm working in driver assistance,…
People dont usually say it but the goal of ISO26262 is to build something that kills people at a rate lower than once in 10^10 hours of operation.
>> Then the tedious work comes. You think through all the parts and safety goals and wonder: If something goes wrong here, would it violate the safety goal? If you don't find a good argument, then you need to build some mechanism to prevent it from going wrong.
Your team might find fault tree analysis useful - I recommend finding someone familiar with the methodology. A good introduction is NUREG 0492 which you can probably find as pdf.
Re: The Three Levels of Software Safety
#9The level of safety depends on the context the software will be used.
Probably the highest level of safety would pertain to situations like the Apollo 11 mission https://github.com/chrislgarry/Apollo-11
But there are other situations where you don’t need such a high level of software safety. eg when prototyping or exploring different possibilities.
Re: The Three Levels of Software Safety
#10Anyone interested in these ideas may also want to take a look at https://mitpress.mit.edu/books/engineering-safer-world and http://mit.edu/psas — both are great resources for learning more.