People’s freedom jeopardised by new software adopted by California’s courts • Hacker News Board

Re: People’s freedom jeopardised by new software adopted by California’s courts

#71

Post by eponeponepon » Wed, Nov 30, 2016, 6:12 PM UTC

briandear wrote: post #54

eponeponepon wrote: post #11

I expect the truth of this lies somewhere in the outsourcing industry. It has every smell of miscommunicated requirements and half-assed implementation. If I'm right, I doubt it'll ever be admitted to though.

I wish I could upvote this a hundred times. I am currently dealing with an Indian dev shop and the code has been atrocious -- as if they didn't even read the requirement. Doing a very basic Stripe integration has taken nearly a week and it was still incorrect. I could provide days of examples.

The people writing your code have certainly not read the requirements. They'll each be working on one single aspect, and focussing entirely on fulfilling some knocked-together unit tests written by someone who might've read the requirements, but only with a view to working out how to make as much existing code as possible from the last customer fit into your project.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#72

Post by jancsika » Wed, Nov 30, 2016, 6:14 PM UTC

noir_lord wrote: post #19

Software is currently in a strange place legislatively, in the 18th century Civil Engineering exploded (sometimes literally) and the number of disasters went up in radically (for an interesting case https://en.wikipedia.org/wiki/Tay_Bridge_disaster ), over time professional standards bodies grew alongside the maturing industry to both ensure that people working in the industry where adequately trained and adequately…

Position one: implement professional-training program and whistleblower-protections for employees of the out-of-state companies writing the proprietary software for CA court system

Position two: require court system software to be open source

You might reply that open source is no panacea, and that's true. But the vulnerabilities in current software are wholly different than the ones in civil (physical) engineering-- both your imagined "professional grade" proprietary court app and my imagined open source court app will have critical vulnerabilities. The question is what happens when they are found. I'd like to read a blog entry complete with screen captures by a security expert who is running the exact same software, posted after the exploit has been responsibly reported and addressed. I do not want to read an opaque statement from a company blaming its users, and I don't see how even a set of strong, effective software engineering guidelines is going to get you anything else in this case.

Edit: I said "vulnerabilities", but the logic applies equally to usability issues, data corruption, etc.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#73

Post by awinter-py » Wed, Nov 30, 2016, 6:16 PM UTC

Separate from the question of software liability is the court's responsibility to not make egregious errors.

Historically, courts have been unwilling to assign themselves blame for screwing up. Judicial immunity is untouchable in american case law because it's hard to find a judge willing to rule against it.

The standard of due process is high for 'life and limb' cases but low for misdemeanors & traffic violations. When Fixed discovered that most SF parking tickets are challengeable, SF didn't fix the problem -- they turned off their fax machine to make it harder to challenge tickets. https://techcrunch.com/2015/10/12/fixed-the-app-that-fixes-y...

NYC issued fake parking tickets to the tune of hundreds of thousands of dollars. http://iquantny.tumblr.com/post/144197004989/the-nypd-was-sy...

Jurisprudence doesn't have a concept of 'bulk miscarriage of justice'. You could put NSA surveillance in this camp too. There's a star trek line about genocide which says 'we have no law to fit your crime'. That's where we are with petty crimes mishandled in bulk.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#74

Post by threatofrain » Wed, Nov 30, 2016, 6:16 PM UTC

Perhaps I'm crazy, but isn't a $5 million contract a bit too low for an overhaul of the California justice software system?

Business-wise there's going to be a lot of things to cut, pushback against government asks for software, and a very skeletal plan for maintenance mode. I sometimes wonder how government models the businesses they do work with, or whether they work as hard as businesses in modelling the other side.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#75

Post by noir_lord » Wed, Nov 30, 2016, 6:17 PM UTC

acbabis wrote: post #63

noir_lord wrote: post #21

Earlier quoted context omitted.

> People writing software need to take responsibility for how the software is used There is an old joke that it's almost impossible to get someone to understand a problem that would result in a threat to their livliehood. I don't think people need to "take responsibility" I think organisations need to be made to take responsibility. In the UK when you purchase a physical good from the retailer the warranty for that g…

I'm not saying it's not a good idea, but that system's gonna be really tough on the software industry. Consider what would happen when a famous third-party API has security flaw. Most third-party libraries' licenses have an "as-is" clause and even mature third-party software has exploits sometimes (e.g. Java applets). If you wanted to use a library in a project, you would have to - a) read the entire thing to see if…

I own one of those small shops so I'm not unsympathetic but few other industries get away with 'as-is' disclaimers,

Can you imagine it in the automotive,aeronautical, transport etc?

The present system is a pass the parcel of blame with no one taking any responsibility, tonnes of shit, insecure systems written and designed poorly.

Eventually something will give.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#76

Post by noir_lord » Wed, Nov 30, 2016, 6:22 PM UTC

jonathankoren wrote: post #64

treehau5 wrote: post #55

Earlier quoted context omitted.

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanis…

No we don't have that distinction. We have places where outsiders, typically from heavily regulated industries like finance, like medicine, like areospace, have imposed testing and validation procedures and best practices onto the software industry. Left to our own devices, we get IoT bullshit like Nest thermostats that randomly turn the temperature down because they turn off[0], security exploit via unneeded running…

Completely agree with all of this and it has to be addressed soon.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#77

Post by Terr_ » Wed, Nov 30, 2016, 6:29 PM UTC

jdietrich wrote: post #58

tener wrote: post #32

Earlier quoted context omitted.

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they…

>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to…

> Nuremberg defence

Except we're not just talking about clearly malicious or fraudulent software with sins of commission. The bigger issue is cutting corners, taking shortcuts, and individual weaknesses that break the whole system.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#78

Post by mdpopescu » Wed, Nov 30, 2016, 6:35 PM UTC

dispose13432 wrote: post #70

noir_lord wrote: post #19

Software is currently in a strange place legislatively, in the 18th century Civil Engineering exploded (sometimes literally) and the number of disasters went up in radically (for an interesting case https://en.wikipedia.org/wiki/Tay_Bridge_disaster ), over time professional standards bodies grew alongside the maturing industry to both ensure that people working in the industry where adequately trained and adequately…

The threat model of code is different than that of bridges, and legislation won't help. Civil engineers have to make sure that bridges won't naturally collapse. Software "engineers" have to ensure that no one will break their code, from the privacy and anonymity of their house, with millions of dollars in payoffs. If bridges held millions of dollars and would be accessible from around the world, they would be breakin…

Agreed. "Programming Satan's Computer" is a good paper on the subject of code running under constant attack.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#79

Post by Spivak » Wed, Nov 30, 2016, 6:40 PM UTC

jdietrich wrote: post #58

tener wrote: post #32

Earlier quoted context omitted.

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they…

>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to…

> Befehl ist Befehl. Blaming management is literally the Nuremberg defence.

This isn't really about shifting blame or absolving responsibility. It is everyone's responsibility to ensure that safety and the public good are put first. If developers don't have the ability to say no to their management because there's no institution backing up that decision then we are to blame.

I know it's a cynical world view but management will trade just about anything for profits and employees will compromise their principles for their livelihood. Hoping that this system of perverse incentives will correct itself internally using only the individual employee's right to resign far too hopeful for my taste.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#80

Post by nine_k » Wed, Nov 30, 2016, 6:41 PM UTC

dispose13432 wrote: post #70

noir_lord wrote: post #19

Software is currently in a strange place legislatively, in the 18th century Civil Engineering exploded (sometimes literally) and the number of disasters went up in radically (for an interesting case https://en.wikipedia.org/wiki/Tay_Bridge_disaster ), over time professional standards bodies grew alongside the maturing industry to both ensure that people working in the industry where adequately trained and adequately…

The threat model of code is different than that of bridges, and legislation won't help. Civil engineers have to make sure that bridges won't naturally collapse. Software "engineers" have to ensure that no one will break their code, from the privacy and anonymity of their house, with millions of dollars in payoffs. If bridges held millions of dollars and would be accessible from around the world, they would be breakin…

Compare it to automotive engineering.

Cars are designed with an assumption that they will be sometimes mishandled, tipped over, crashed into things, run into bodies of water, set on fire, etc. In these circumstances cars should behave as to best preserve humans' safety, both inside (passengers) and outside (pedestrians).

This leads to some serious design decisions on deeper level, e.g. making the engine go under the rest of the car on a frontal collision, adding a structural cage around the passenger compartment, etc.

Deeply defensive programming methodologies also exist, from using safer languages and formally proven algorithms to pen-testing the completed setup. Their adoption just costs effort and time, that is, money.