The Three Levels of Software Safety • Hacker News Board

Re: The Three Levels of Software Safety

#11

Post by _urga » Tue, Dec 29, 2020, 5:51 PM UTC

qznc wrote: post #3

For an article which aspires to clarify safety I would expect a good definition of the term. Is it "preventing a system from reaching dangerous states"? In my opinion, you should start with defining what harm to humans you think about. Harm due to privacy leaks? Financial loss due to miscalculations? Locked out by your home security system? Electrocuted by your home security system? I'm working in driver assistance,…

"How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence."

10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?

I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.

Re: The Three Levels of Software Safety

#12

Post by toss1 » Tue, Dec 29, 2020, 10:48 PM UTC

qznc wrote: post #3

For an article which aspires to clarify safety I would expect a good definition of the term. Is it "preventing a system from reaching dangerous states"? In my opinion, you should start with defining what harm to humans you think about. Harm due to privacy leaks? Financial loss due to miscalculations? Locked out by your home security system? Electrocuted by your home security system? I'm working in driver assistance,…

Interesting description, thanks!

>>"decelerate no more than 60km/h in 1.2 seconds" >>"never send signal AebBreakRequest=1 for longer than 1.2seconds"

I'm surprised to see these specified like this. It seems like it would be much more useful and responsive to real world conditions to specify it is G-Load profiles over milliseconds, especially since the effective braking force is dependent on so many conditions of the vehicle & environment.

So, what do I not know about your system and dev environment?

Re: The Three Levels of Software Safety

#13

Post by OnlyOneCannolo » Tue, Dec 29, 2020, 11:28 PM UTC

lixtra wrote: post #2

> Safety is ultimately about preventing a system from reaching dangerous states. This is a fallacy. Whether a state is dangerous or not often depends on the environment outside the system and usually cannot be totally foreseen by the engineer. I happily damage my engine if I can drive out of the dessert. Depending on case I’m happy if files can be undeleted - or not. But of course I appreciate a warning why the state…

Every safety analysis has a system boundary and a definition of "loss". In your example, the topmost system would also include you as a subsystem, and the loss you're trying to prevent is injury to yourself. Getting stranded is a hazard that could result in that loss. A mitigation could be to allow you to damage the vehicle subsystem to protect the person subsystem. A formal safety program would try to avoid process escapes such as overlooking a scenario like that. You might still decide not to design for it, but at least you consciously made that decision.