Programmers who want to change how we code before catastrophe strikes • Hacker News Board

Re: Programmers who want to change how we code before catastrophe strikes

#81

Post by throwaway020217 » Wed, Sep 27, 2017, 7:51 PM UTC

always_good wrote: post #31

chroem- wrote: post #24

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when…

>The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Well, it's a whole ecosystem problem. Most of the places I've worked, security was just supposed to be something you learned and credentialized on your own, and no time or resources were deliberately allocated to it. In fact, you're likely seen as an underperformer if you slow down to get things right…

In professional engineering the accountability comes with a stamp, and a professional ethics requirement to report up the chain if a critical problem has been found that could impact public safety. This includes whistleblowing if management ignores the problem.

There are clear guidelines as to what types of products and services require a p.eng.'s stamp. There are also many professional engineers out there who's stamp has never touched ink. For example, the engineers drafting and churning out spreadsheets in the engineering sweatshops aren't accountable for the safety of the product, you'll have one or more engineers providing their stamp for that group's work.

If we were to implement a similar model on software development, most developers would not have personal liability for work they do e.g. for a company or open source project. What is needed is better classification of software products/activities that require a software engineer's stamp[1], and a corresponding professional association.

So no, developers should not be accountable. However, certain safety-critical products - where safety can be defined in various ways expanding the current relatively small set of NASA, aviation and health-care software - need engineers overseeing design, development, and QC to be accountable.

And yes, this means that the self-taught developer, no matter how talented, will not be able to release certain types of software without some sort of oversight.

[1]As a side-note - not everyone can call themselves an 'engineer' in other fields without the proper credentials and professional membership. I see developers calling themselves software engineers all the time though they don't have the chops. This would, of course, need to change.

Updated to change asterisk to brackets

Re: Programmers who want to change how we code before catastrophe strikes

#82

Post by [deleted] » Wed, Sep 27, 2017, 7:53 PM UTC

tmnvix wrote: post #3

> "Programmers were like chess players trying to play with a blindfold on—so much of their mental energy is spent just trying to picture where the pieces are that there’s hardly any left over to think about the game itself." This is an insightful description of the biggest mental challenge I face when programming.

[deleted]

Re: Programmers who want to change how we code before catastrophe strikes

#83

Post by IncRnd » Wed, Sep 27, 2017, 7:53 PM UTC

wpietri wrote: post #74

chroem- wrote: post #24

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when…

Motivation seems far from the only difference. If we divide making into "builds with atoms" and "builds with bits", both sorts of things are generally unregulated. It's when you get into "builds life-critical things with atoms" that you see significant regulation. If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't. I agree that soci…

> If you build a house or an office building, your engineering is regulated. If you build a doghouse or an office filing cabinet, it isn't.

A filing cabinet must be fit for the purpose it is used, or there will be liability incurred. OSHA would frown on keeping hazardous chemicals in your filing cabinet. Software should also be fit for the purpose it is marketed and used.

There are many kinds of paper, but paper being used improperly, by storing a list of credit card numbers on it, is quite similar to how many pieces of software are written.

Re: Programmers who want to change how we code before catastrophe strikes

#84

Post by Bartweiss » Wed, Sep 27, 2017, 7:55 PM UTC

haskellandchill wrote: post #59

vtange wrote: post #49

So we need to write more code to make fancy Photoshop-like editors for the average-Joe programmer who can't see the big picture? That just adds more to the code issue - even Photoshop has bugs you know. The fact that "Few programmers write even a rough sketch of what their programs will do before they start coding" is a soft-skills/experience issue that isn't specific to software. I too can use Photoshop to draw logo…

No, we need to use logic and set theory and provide tools for visualizing the implications of our rules and checking correctness of desired properties. There's a strong history and lots of good people working on these things but it's tough to get our message out to working programmers. This article helps but based on the comments we've got a lot of perception work to do :)

There are definitely people making a real effort here, and I appreciate you. But I can't shake the feeling that the reason your task is so hard is that everyone before you has been selling snake oil. "Visual programming", "human-readable software languages" and so on are all just ways of saying "crippled tools".

It's not an accident that the examples in the article were WYSIWYG editors, Photoshop, Squarespace, and Mario. All of those things flatten neatly into two dimensions, and their terminal form is visual. The visualized code is the product.

Meanwhile similar initiatives for software in general are almost always nonsense. There was a pretty compelling TED talk* about visual tools for cybersecurity, shared by lots of people I know. Only one problem: none of the fancy tools showcased work, or will work. They're operating backwards from a known answer. Like most programming tools, they collapse right where they're needed most, stopping at the same edge cases that cause these problems.

More broadly, it seems like the existing tools for known-safe software run exactly the opposite direction from Bret Victor's vision. Correctness proofs, exhaustive test suites, reversible debuggers, and so on are far from non-code, but they're what we use where reliability matters most.

I can appreciate that "code everything like the space shuttle" is hopeless, and I would love to see breakthroughs on easier correctness checking. But right now, it does feel like an attempt to push tools that won't work when they're needed.

*TEDx, but selected by TED for special featuring: https://www.ted.com/talks/chris_domas_the_1s_and_0s_behind_c...

Re: Programmers who want to change how we code before catastrophe strikes

#85

Post by tasuki » Wed, Sep 27, 2017, 7:58 PM UTC

I'm surprised understanding the domain hasn't been mentioned. Whether I am developing for someone else or for myself, it turns out misunderstanding/misrepresenting the domain is the most common source of trouble.

If your understanding of the domain isn't thorough, is TLA+ going to be much help?

Re: Programmers who want to change how we code before catastrophe strikes

#86

Post by AnimalMuppet » Wed, Sep 27, 2017, 8:00 PM UTC

haskellandchill wrote: post #59

vtange wrote: post #49

So we need to write more code to make fancy Photoshop-like editors for the average-Joe programmer who can't see the big picture? That just adds more to the code issue - even Photoshop has bugs you know. The fact that "Few programmers write even a rough sketch of what their programs will do before they start coding" is a soft-skills/experience issue that isn't specific to software. I too can use Photoshop to draw logo…

No, we need to use logic and set theory and provide tools for visualizing the implications of our rules and checking correctness of desired properties. There's a strong history and lots of good people working on these things but it's tough to get our message out to working programmers. This article helps but based on the comments we've got a lot of perception work to do :)

How long have you worked as a professional programmer? Because you sound like a just-out-of-school idealist with an idea of the "right way to do it". That ideal tends to get blunted after a decade or two of experience in the real world.

In particular: Set theory? I don't want to program using set theory for the same reason that I don't want to program using octal - it's the wrong level of abstraction for what we're actually doing, no matter whether it's formally equivalent or not.

Tools for checking the correctness of desired properties? I can agree with you there; better tools for doing so would be useful. So far, the problem tends to be that the tools are less generally useful than the tools' authors and proponents think they are.

Re: Programmers who want to change how we code before catastrophe strikes

#87

Post by peoplewindow » Wed, Sep 27, 2017, 8:00 PM UTC

bsaul wrote: post #9

Great article, but it leaves me with one question : what's the difference between good code generating systems (the ones mentionned, that you'd use to build critical system), and the horrors that we saw back in the time with wysiwig HTML editors such as dreamweaver ( to the point where nobody today would dare to write an html website with something other than a text editor) ? The second problem seems a lot easier to…

Dreamweaver was actually famous at the time for creating clean and readable HTML. It's FrontPage you're thinking of.

Re: Programmers who want to change how we code before catastrophe strikes

#88

Post by SilasX » Wed, Sep 27, 2017, 8:02 PM UTC

brians wrote: post #72

skywhopper wrote: post #58

This article is silly in a lot of ways. The real problem of software engineering is not "how do you prove this code follows algorithm X exactly?" It's "how do you know what algorithm X needs to be in sufficient detail to implement it?" It's "when you realize you were wrong about algorithm X, how do you change your existing, working code to implement the new algorithm X-prime, without interruption?" It's "what do we d…

The article's not silly. Leveson and Hamilton's works cover that explicitly. Start with Engineering a Safer World's Intent Specifications, move on to Safeware, and look at the attention that goes into writing the right TLA+ specs. Lists like "Government regulations, flaky hardware…" look a lot like the upper levels of the hierarchical control systems contemplated there. It didn't make it into a pop science piece, but…

It sounds like you're both in agreement, which would support the criticism of the article, which hides the (addressing of the) very points the parent was raising, and gives the impression that it only solves the very academic version of the problem rather than the real-world software snags.

Re: Programmers who want to change how we code before catastrophe strikes

#89

Post by ScottBurson » Wed, Sep 27, 2017, 8:03 PM UTC

tasuki wrote: post #85

I'm surprised understanding the domain hasn't been mentioned. Whether I am developing for someone else or for myself, it turns out misunderstanding/misrepresenting the domain is the most common source of trouble. If your understanding of the domain isn't thorough, is TLA+ going to be much help?

> If your understanding of the domain isn't thorough, is TLA+ going to be much help?

I suspect it could, actually, because it lets you formalize and work with the implications of the understanding you do have without getting bogged down in the details of actual coding. Seems like this could give you the opportunity to debug your mental model much earlier in the process. (I confess I haven't actually tried TLA+, but I plan to.)

Re: Programmers who want to change how we code before catastrophe strikes

#90

Post by Bartweiss » Wed, Sep 27, 2017, 8:04 PM UTC

emodendroket wrote: post #55

Bartweiss wrote: post #52

Earlier quoted context omitted.

It's not like reliable, secure software doesn't exist. It's just limited to spaces where people are willing to use years-old tech, forgo new features, and pay fortunes for auditing. Air travel, industrial machinery, medicine, and so on. Almost like the cost-benefit actually doesn't support making the development of trivial software ten times slower and more expensive... If there's a problem here, it's business extern…

I mostly agree, but this article cites the national 911 software, which arguably should be treated as a critical system, failing, and some medical devices whose software killed people.

Yeah, fair enough - I've mostly been responding to too-general comments, the article is more specific.

The 911 thing seems like an obvious issue that absolutely should have been caught by a rigorous development system; it was treated like a digital phone system instead of a life-saving service. The Therac bug sort of feels like a different story, because that bug led to IEC 62304 - it's older than the practices which would now prevent it. They're both critical systems that were written without due care, though.