A sharper line between critical and non-critical software would help. Only some software needs to work well.
What Should We Do to Prevent Software from Failing?
61–70 of 128 posts
Re: What Should We Do to Prevent Software from Failing?
#62"Consider the construction industry, which has had formal standards in place for decades."
The construction industry has existed, in some form, for centuries. It has had time to develop these formal standards. Software is much younger, and the "formal standards" it should follow are still in flux.
Re: What Should We Do to Prevent Software from Failing?
#63Earlier quoted context omitted.
None of the above changes the discussion. It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers. It's not that their designs are made flawless, it's that (in theory) the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in…
Perhaps one of the reasons for this is that software doesn't allow humans any way of leveraging our evolved "danger detectors". It's hard to make a safe building, and it's hard to make safe software. The difference is it's (often) incredibly obvious that a building isn't safe the moment you step into it. In software it's sometimes pretty much impossible to tell the difference between a stone temple and a house of car…
> The difference is it's (often) incredibly obvious that a building isn't safe the moment you step into it.
This is just not true. If anything, our "danger detectors" probably give off an equal amount of false positives and false negatives when judging the structural integrity of objects we're unfamiliar with. And the same holds true for code bases, which is exactly why it makes sense to hold software to a similar standard as physical engineering.
Re: What Should We Do to Prevent Software from Failing?
#64The analogy between software "engineering" and structural engineering is used a lot to point out the failures of software. I think this analogy is a category mistake caused entirely by the misapplication of the word "engineer" to software developers. We have already solved large areas of common classes of bugs at the language level, through strong static typing, memory safety and safe concurrency (Rust is the vanguar…
None of the above changes the discussion. It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers. It's not that their designs are made flawless, it's that (in theory) the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in…
A key difference between physical engineering and software engineering is that the economics of over-engineering to improve safety margins are wildly different. For physical engineering, the costs of substantially decreasing failure risk (e.g. via increasing strength, different materials, more complex design, etc) is approximately linear. For software engineering, that cost is something closer to quadratic or worse.
This is intrinsic to the differing nature of the problems. Verifying the safety of a bridge design is computationally trivial, verifying the safety of non-trivial software is NP-Hard. Safety is so inexpensive to add in physical engineering designs that we invest a lot of effort removing incidental unnecessary safety.
Re: What Should We Do to Prevent Software from Failing?
#65The analogy between software "engineering" and structural engineering is used a lot to point out the failures of software. I think this analogy is a category mistake caused entirely by the misapplication of the word "engineer" to software developers. We have already solved large areas of common classes of bugs at the language level, through strong static typing, memory safety and safe concurrency (Rust is the vanguar…
None of the above changes the discussion. It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers. It's not that their designs are made flawless, it's that (in theory) the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in…
It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers... the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in the field, amount to a very very high success rate.
It's not only hard, but the cost/benefit (or the large penalties of catastrophic failure) works out so that it's worth it to go to that amount of effort. The same amount of effort doesn't go into building a backyard garden shed. I'll grant that we've gotten to the point where the cost/benefit analysis demands that we go to that amount of effort in critical situations.
Re: What Should We Do to Prevent Software from Failing?
#66Speaking as someone who builds mission critical software, licensing is not the answer to this problem. Occupational licensing only serves to raise barriers to entry and crowd out competition without raising quality. To put my opinion bluntly, Bernie Madoff had a license. The things you should do to prevent catastrophic failure: - Reduce your attack surface as much as possible. - Automate your infrastructure, human in…
> Occupational licensing only serves to raise barriers to entry and crowd out competition without raising quality. To put my opinion bluntly, Bernie Madoff had a license.
Almost every country in the world established at one point or another some kind of licensing for various professions. I think the barrier of "this is all bullshit" is a bit higher than one bad apple for something found to be a good idea by so many people. Do you have some supporting arguments for your position?
Re: What Should We Do to Prevent Software from Failing?
#67The problem with software is that, for almost all software, the only specification for what it is supposed to do is discovered as the software is written. Nobody knows all the data, the data semantics, its dependencies, what output is really required and how that it to be computed from that data. Virtually all cases, and certainly all software I've ever been involved with, all this is discovered on the fly during the development effort. We even encode this into our processes, after all one of the main reason agile and similar processes exist is the understanding that we need to iterate to discover what the spec should be.
If a full specification is truly known up front and validated, then building software that works correctly to that specification is a well understood problem with well understood processes (sure, much more time consuming and expensive than regular software engineering, but that is a different category of issue)
But look at the Boeing 737-Max8 problem: the issue doesn't appear to have been the implementation of the software, the problem seems to have been that the specification itself was poorly constructed for business reasons. And now software gets the blame? Sigh.
Re: What Should We Do to Prevent Software from Failing?
#68Earlier quoted context omitted.
None of the above changes the discussion. It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers. It's not that their designs are made flawless, it's that (in theory) the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in…
A key difference between physical engineering and software engineering is that the economics of over-engineering to improve safety margins are wildly different. For physical engineering, the costs of substantially decreasing failure risk (e.g. via increasing strength, different materials, more complex design, etc) is approximately linear. For software engineering, that cost is something closer to quadratic or worse.…
> verifying the safety of non-trivial software is NP-Hard.
While true, this depends very much on the design (or lack thereof) of the software system. Many of our software systems are more complex than necessary because they weren't designed or the designs weren't considered carefully enough. When the software is allowed to grow and accumulate masses of code virtually unchecked, it becomes computationally intractable to continue to verify the system.
Much of the software involved in aircraft, specifically, can be reduced to very simple designs and implementations where the necessary safety analysis is feasible.
The present complexity is often in the accumulation of legacy defects and corrections (fixing the right problem in the wrong place) or trying to fix hardware problems with software. I'm not 100% pro-rewrite, but many avionics software groups (that I've encountered) refuse any rewrites on certification grounds but end up with something unbearably complex as a result (simulating hardware databuses from the 1980s in software on a 1 GHz processor when the system was initially targetting a 16 MHz processor).
Re: What Should We Do to Prevent Software from Failing?
#69I have seen a number of critical systems with frankly masturbatory code. Well designed and structured code is a rarity.
Re: What Should We Do to Prevent Software from Failing?
#70Earlier quoted context omitted.
None of the above changes the discussion. It's a very elaborate way of saying "writing correct software is hard". You know what's also hard to do correctly? Designing airplanes. Building skyscrapers. It's not that their designs are made flawless, it's that (in theory) the sheer volume of man-hours spent pouring over every minute detail of them, incorporating lessons learned from every failure that's ever happened in…
A key difference between physical engineering and software engineering is that the economics of over-engineering to improve safety margins are wildly different. For physical engineering, the costs of substantially decreasing failure risk (e.g. via increasing strength, different materials, more complex design, etc) is approximately linear. For software engineering, that cost is something closer to quadratic or worse.…
Try to solve an aerospace problem with this "simple" approach. No, you don't get to add strength willy-nilly, or just swap materials on a hunch. You're working with safety factors which sometimes are less than 1 (come back for references), and those physical properties of materials are notoriously stubborn to changes. You know, some Richard Feynman once quipped "nature can't be fooled, dammit" - that extends to "water boils at 273.16K full stop (standard conditions), no, you're not going to change that".
Yes, you can often do more complex design. Sometimes though it costs you centuries to choose a good combination of existing physical properties (architecture), and sometimes it's "just" decades (Gravity Probe B). Wanna propose similar approach to software? :)