People’s freedom jeopardised by new software adopted by California’s courts • Hacker News Board

Re: People’s freedom jeopardised by new software adopted by California’s courts

#101

Post by CamperBob2 » Wed, Nov 30, 2016, 9:14 PM UTC

hackuser wrote: post #87

jdietrich wrote: post #58

Earlier quoted context omitted.

>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to…

> Pressure needs to come from users, but also from developers It needs to come from government especially. It's a basic function of government to protect the public > How would we react if this story was about a plane crash or a nuclear accident caused by negligence? If there were no government laws penalizing this outcome and regulations preventing it, I'd be horrified.

There are plenty of regulations covering software used in aircraft and nuclear power control. Move on to the next moral panic.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#102

Post by alistairSH » Wed, Nov 30, 2016, 9:16 PM UTC

tener wrote: post #91

jdietrich wrote: post #81

Earlier quoted context omitted.

There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track dow…

You are missing the big picture here. The key phrase is: "risk management". Anything that your are doing (or NOT doing) have inherent risks. You should manage those risks. If risk realises you will be losing money (in various ways, including litigation) or other assets (such as reputation, talented people, market share etc.). You can lower your exposure to the risk generally by spending money on it. You can hire more…

If you could go to out of business or to prison because the software you have delivered is faulty

And that's the problem. If a civil engineer designed a bridge that maimed hundreds, he would lose his license. There is no license to be lost for software developers.

What will be the cost to the company that built this shoddy justice system software? Will anybody go to jail or be personally held accountable? I doubt it (I can't think of cases where a developer has been personally responsible).

Re: People’s freedom jeopardised by new software adopted by California’s courts

#103

Post by sedachv » Wed, Nov 30, 2016, 9:57 PM UTC

dispose13432 wrote: post #84

nine_k wrote: post #80

Earlier quoted context omitted.

Compare it to automotive engineering. Cars are designed with an assumption that they will be sometimes mishandled, tipped over, crashed into things, run into bodies of water, set on fire, etc. In these circumstances cars should behave as to best preserve humans' safety, both inside (passengers) and outside (pedestrians). This leads to some serious design decisions on deeper level, e.g. making the engine go under the…

>Cars are designed with an assumption that they will be sometimes mishandled But not maliciously. If you run your car at another, the other car will break. I would compare it to lock-making. Despite thousands of years of lock-making, they _still_ get broken, and there's nothing to be done about it. >Deeply defensive programming methodologies also exist, from using safer languages and formally proven algorithms to pen…

> It's not just money. It's paperwork, bureaucracy, and ultimately not effective. Practically, the only way to ensure safety is to take down the internet.

https://logfall.wordpress.com/all-or-nothing-fallacy/

We can and should be making incremental improvements to technology. Continuing to use methods that are known to be faulty is gross negligence in the legal sense: https://en.wikipedia.org/wiki/Gross_negligence

We do not see enough lawsuits around software negligence today because companies are under no obligation to release source code - there is no way to tell how or why or even whether software is defective. This was a major issue in the investigation of the Toyota unintended acceleration scandal. The investigation uncovered major, obvious code quality problems, ultimately concluded the fault was caused by software (something Toyota denied, and without the investigation, consumers never would have learned), and I hope becomes a precedent: http://www.safetyresearch.net/blog/articles/toyota-unintende...

IMO if poor coding practice is gross negligence, it follows that failing to reveal the source code constitutes intent: poorly developed closed-source software is criminal negligence.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#104

Post by tener » Wed, Nov 30, 2016, 10:10 PM UTC

alistairSH wrote: post #102

tener wrote: post #91

Earlier quoted context omitted.

You are missing the big picture here. The key phrase is: "risk management". Anything that your are doing (or NOT doing) have inherent risks. You should manage those risks. If risk realises you will be losing money (in various ways, including litigation) or other assets (such as reputation, talented people, market share etc.). You can lower your exposure to the risk generally by spending money on it. You can hire more…

If you could go to out of business or to prison because the software you have delivered is faulty And that's the problem. If a civil engineer designed a bridge that maimed hundreds, he would lose his license. There is no license to be lost for software developers. What will be the cost to the company that built this shoddy justice system software? Will anybody go to jail or be personally held accountable? I doubt it…

If you are going to put to jail a single developer, you may as well put to jail the entire company. That in turn wouldn't be practical or fair.

It isn't the developer who has signed the contract to deliver the faulty piece of software that started this whole discussion.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#105

Post by sedachv » Wed, Nov 30, 2016, 10:26 PM UTC

ptaipale wrote: post #39

mtgx wrote: post #33

Earlier quoted context omitted.

> “With the old system, it took maybe one or two clicks to complete a process,” she said. “Now it takes 25 clicks, and there are drop-down boxes and all of that.” > Because the system is so unwieldy, clerks are unable to enter data in the courtroom, she said, so that burden has fallen on other office workers. It’s created a backlog of more than 12,000 files that have not been uploaded — and that number is growing by…

However, the fault probably does not lie in someone writing bad software; it's more about the authorities themselves defining a bad workflow when specifying the system. I say this based on just a bit of personal experience in working with authorities and their mode of operation (in Europe but I expect US is not that different): it's often about covering one's ass by insisting on lots of checks and balances, and the w…

> However, the fault probably does not lie in someone writing bad software; it's more about the authorities themselves defining a bad workflow when specifying the system.

> This looks like fundamentally a public sector process issue. They'll improve it though.

The better, "old system" was ordered by the same government office - two different results from the same "public sector process" should give a hint that maybe that is not the root cause of the problem. At the same time, since your comment attacks the public sector without providing alternatives, I can only assume that you are implying that a private company would handle this better - are you proposing to privatize the courts?

Re: People’s freedom jeopardised by new software adopted by California’s courts

#106

Post by kag0 » Wed, Nov 30, 2016, 10:36 PM UTC

dkarl wrote: post #17

Similar problems have been reported in Tennessee and also in Indiana - where prosecutors have had a perhaps more troubling issue of inmates being mistakenly released early. It seems backwards to describe this as more troubling.

Keep in mind this is a UK site. People are guilty until proven innocent, so from that perspective guilty people going free may indeed seem more troubling.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#107

Post by sedachv » Wed, Nov 30, 2016, 10:39 PM UTC

acbabis wrote: post #63

noir_lord wrote: post #21

Earlier quoted context omitted.

> People writing software need to take responsibility for how the software is used There is an old joke that it's almost impossible to get someone to understand a problem that would result in a threat to their livliehood. I don't think people need to "take responsibility" I think organisations need to be made to take responsibility. In the UK when you purchase a physical good from the retailer the warranty for that g…

I'm not saying it's not a good idea, but that system's gonna be really tough on the software industry. Consider what would happen when a famous third-party API has security flaw. Most third-party libraries' licenses have an "as-is" clause and even mature third-party software has exploits sometimes (e.g. Java applets). If you wanted to use a library in a project, you would have to - a) read the entire thing to see if…

> Imagine paying a monthly subscription fee just so you can offset the liability when Angular has a security hole. This would run a lot of small shops out of business.

Surgeons are not liable for, for example, hip replacement implants from a decade ago that are now known to have problems, but they would be responsible if they continued using the implants after they were recalled. They also purchase malpractice insurance. How is this any different than following security advisories and purchasing liability insurance?

Re: People’s freedom jeopardised by new software adopted by California’s courts

#108

Post by TheSpiceIsLife » Wed, Nov 30, 2016, 11:01 PM UTC

Retric wrote: post #92

tener wrote: post #86

Earlier quoted context omitted.

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some cra…

This comes back to how software is designed and built. On it's own a single bad weld should never endanger a bridge, but we accept a single mistake can bring down software. At it's core that's why software fails so often AND that's what we need to fix.

On all the bridges we build every weld is non-destructively tested before the item is painted and shipped out.

Magnetic particle, and penetrant die testing.

I don't understand the term 'Software Engineer'. Still feels like the Wild West me.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#109

Post by flukus » Wed, Nov 30, 2016, 11:08 PM UTC

Retric wrote: post #92

tener wrote: post #86

Earlier quoted context omitted.

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some cra…

This comes back to how software is designed and built. On it's own a single bad weld should never endanger a bridge, but we accept a single mistake can bring down software. At it's core that's why software fails so often AND that's what we need to fix.

A single bad weld is isolated, a single bug can propagate a lot of bad data. Software crashing isn't the same as a bridge crashing, in many cases crashing is the best thing that can happen.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#110

Post by CodeMage » Wed, Nov 30, 2016, 11:15 PM UTC

tener wrote: post #40

CodeMage wrote: post #35

Earlier quoted context omitted.

I would amend that to "don't put the blame solely on the developers". I've seen way too many developers whose mentality is "ship it, it seems good enough, we'll fix any bugs as we go".

If the company culture is supporting this attitude, then sure - you will have developers saying that. But it isn't the developers that create that culture. This is something the management is responsible for.

People aren't computers, they don't just run the company culture program in their brains. Sure, developers are influenced by company culture, but they contribute to it as well.

At one end of the spectrum, you have junior devs, fresh out of school, highly susceptible to the effects of company culture. At the other, you have senior devs with more than a decade of experience, who do a great deal to set the tone and shape the culture.

Thank about what kind of cultural influence you get from a senior dev who has accumulated more than ten years of experience of moving fast and breaking things. That's a lesson that I found very hard to learn: just because someone has lots of experience and skill and knowledge, doesn't mean they are better developers.

Mind you, I'm not saying the blame for the state of our industry lies solely with devs. I agree with you that it's the management that has the last word, but that doesn't absolve the rest of us of all responsibility.