Live data from Hacker News

Programmers who want to change how we code before catastrophe strikes

theatlantic.com

61–70 of 274 posts

Re: Programmers who want to change how we code before catastrophe strikes

#61

Earlier quoted context omitted.

it's like blaming the assembly line worker for a badly designed car. Software engineers don't solve problems, they implement solutions someone else thinks will solve a problem.

Was thinking about this the other day: in big companies SWE individual contributors are the new line workers. Large enterprise ends up trying to manage them with business metrics goals without much visibility into the process issues that hamper overall quality or the production efficiency. Has anybody thought to apply the Toyota Production System in Engineering management? https://en.wikipedia.org/wiki/Toyota_Product…

Yes, a whole lot of people have thought of that.

Re: Programmers who want to change how we code before catastrophe strikes

#62

This article is silly in a lot of ways. The real problem of software engineering is not "how do you prove this code follows algorithm X exactly?" It's "how do you know what algorithm X needs to be in sufficient detail to implement it?" It's "when you realize you were wrong about algorithm X, how do you change your existing, working code to implement the new algorithm X-prime, without interruption?" It's "what do we d…

It's not a one size fits all thing. The majority of startups won't need these methods, think software for examples in the article: control systems, critical infrastructure, etc. It's fine to have different approaches for different classes of software. Raising awareness of these tools is critical because most software engineers haven't put much thought into them.

Re: Programmers who want to change how we code before catastrophe strikes

#63

Earlier quoted context omitted.

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompte…

Most of the article, and the comment you're replying to, wasn't about hackers, though. It gave the example of people dying because the Therac machine or a Toyota car responded wrongly to non-malicious, normal user input, like a "bridge collapsing unprompted" in normal use.

Yeah, I agree about the article. I wouldn't offer this criticism to the main essay - fundamental bugs in critical systems are absolutely an engineering failure, at least in the sense that we should look for engineering practices to solve them.

But with the post I responded to? I see this idea of "it's because engineers don't care, so give them liability" a lot, and frankly it's stupid. It neglects both cost-benefit differences and any interest in the question of hacking, and then goes on to pretend that you can fix bad systemic pressures by beating up the guys at the bottom. The difficulty of writing good code is not a function of how heavily you punish bad code.

I flatly don't believe that the Therac bug would have been caught by personal liability for the bug; either the code would have been cancelled altogether to avoid liability or the bug would have gone on unabated. Realistically, criminal liability for lethal bugs mostly sounds like a way to drive talented devs away from high-stakes applications and leave those apps to be handled by people desperate enough to run the risk, or by international shops operating outside the reach of prosecution. I don't love the thought of a shop unwilling to use git because it creates personal responsibility. "Oh, we just pass code around on thumbdrives, I can't remember who wrote that critical line."

I'd shrug and let the whole thing go, but the rumblings of "regulate software" keep getting stronger from a lot of directions. I don't really want to wake up in a decade and find out that degree requirements and expensive-but-useless licensing have become mandatory, and it feels like pretending bugs come from lazy devs is another snowflake in that avalanche.

Re: Programmers who want to change how we code before catastrophe strikes

#64
post #24

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when…

>The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation.

Regulation like that tends to follow after a series of disasters causing massive loss of life. Software hasn't yet managed to achieve that.

Re: Programmers who want to change how we code before catastrophe strikes

#65
post #24

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when…

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompte…

You raise an interesting question here. Civil engineering these days is done on computers, mostly with incredibly convoluted FEM software and other simulation tools. All software vendors refuse to take any responsibility for bugs in their software and even block you from using the software if you don't agree on the terms.

Re: Programmers who want to change how we code before catastrophe strikes

#66
post #24

The dangerous unreliability of software is an accountability problem, not a technology problem. Engineering disciplines are regulated so that engineers are held personally and even criminally responsible if they endanger property or lives. Software, being the newcomer, has no such regulation. Instead we have people trying to claim that software deserves a free pass from due diligence because "software is hard," when…

> The only difference is that these other engineers are motivated to be significantly more thorough about their jobs. Hang on. Sure, all engineering has hurdles. But how many engineering projects face regular enemy action? The last time somebody actively tried to compromise a system I was building was two weeks ago. How many engineers live like that? We hold civil engineers responsible when bridges collapse unprompte…

> But let's not pretend that software has the same difficulties as every other type of engineering. Most engineering doesn't face an epidemic of people actively trying to disable safety precautions to hurt people, and a lot of it reliably fails as soon as it does face enemy action. Punishing every engineer who ever gets outwitted (say, by a leaked NSA-developed vulnerability) is absurd.

The issue isn't exactly how software fails, but that software today doesn't properly protect confidentiality and integrity of data, ensure services and data are available without providing oracles, authenticate and authorize correctly, log properly, etc. This is a well known methodology that is consistently ignored in preference to shipping products immediately. That's the issue.

Unexpected things happen, accidentally or maliciously, but care should be taken - the proper kind of non-negligent care called due care. In general, software is not fit for the purpose it was created, appearing to work even though it really doesn't.

Re: Programmers who want to change how we code before catastrophe strikes

#67

Earlier quoted context omitted.

it's like blaming the assembly line worker for a badly designed car. Software engineers don't solve problems, they implement solutions someone else thinks will solve a problem.

Was thinking about this the other day: in big companies SWE individual contributors are the new line workers. Large enterprise ends up trying to manage them with business metrics goals without much visibility into the process issues that hamper overall quality or the production efficiency. Has anybody thought to apply the Toyota Production System in Engineering management? https://en.wikipedia.org/wiki/Toyota_Product…

DevOps takes this as a primary input.

Re: Programmers who want to change how we code before catastrophe strikes

#68
post #13

The article didn't mention it, but I think that Excel is a great example of where "the masses" have learned to do programming. It is visual and the effects of changes are immediate.

The effects of a change may be immediate, but not necessarily immediately obvious. Subtle bugs are a problem in any programmable system.

Yes, and there are plenty of examples of spreadsheet-bugs having significant real-world impact. E.g.,

1. https://www.bloomberg.com/news/articles/2013-04-18/faq-reinh...

2. https://www.cio.com/article/2438188/enterprise-software/eigh...

Re: Programmers who want to change how we code before catastrophe strikes

#69
post #13

The article didn't mention it, but I think that Excel is a great example of where "the masses" have learned to do programming. It is visual and the effects of changes are immediate.

And yet its results aren't any more reliable. They seem less so to me. https://www.bloomberg.com/news/articles/2013-04-18/faq-reinh...

Re: Programmers who want to change how we code before catastrophe strikes

#70
I work in security. People not giving enough shits about the "situation in the field" with respect to how their code will be used, how long it will need to be used and what will need to be done to keep it functional over that time is job security for me.

Some points the author makes I agree with, others I don't. Nothing will change until incentives change.

Post reply on HN