People’s freedom jeopardised by new software adopted by California’s courts • Hacker News Board

Re: People’s freedom jeopardised by new software adopted by California’s courts

#91

Post by tener » Wed, Nov 30, 2016, 7:29 PM UTC

jdietrich wrote: post #81

treehau5 wrote: post #55

Earlier quoted context omitted.

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanis…

There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track dow…

You are missing the big picture here. The key phrase is: "risk management". Anything that your are doing (or NOT doing) have inherent risks. You should manage those risks.

If risk realises you will be losing money (in various ways, including litigation) or other assets (such as reputation, talented people, market share etc.). You can lower your exposure to the risk generally by spending money on it. You can hire more devs, testers, enact better policies, increase auditing etc. You can even buy insurance.

What you are advocating for is myopic. You should decrease your risk exposure in a way that is most efficient. It may be with better software, but often this isn't the case.

What the public can and should do is to work on the other part of the equation: make the failure more costly. You should "make them pay" - literally. If you could go to out of business or to prison because the software you have delivered is faulty, you will really make sure it isn't. Simple as that.

EDIT: a few recent banking regulations are good examples. In general various regulators have demanded that the international banks setup demonstrably independent local companies with appointed heads that are criminally responsible for misdeeds done those local companies. Putting in prisons few developers isn't goint to fix anything - the company will simply hire another few. Threatening the people in power is the way to go.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#92

Post by Retric » Wed, Nov 30, 2016, 7:33 PM UTC

tener wrote: post #86

ThatGeoGuy wrote: post #60

Earlier quoted context omitted.

> Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA ( https://www.apega.ca ) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being abl…

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some cra…

This comes back to how software is designed and built. On it's own a single bad weld should never endanger a bridge, but we accept a single mistake can bring down software. At it's core that's why software fails so often AND that's what we need to fix.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#93

Post by logfromblammo » Wed, Nov 30, 2016, 7:36 PM UTC

jamesvl wrote: post #37

I worked as a programmer for a smaller California county court system for about five years, and have seen something very much like this play out before (both in my county and others). I can't comment on the Tyler product or their training directly; maybe they really are a rock star outfit. But if this is like past attempts, this project has all of of the worst of aspects of software development risks and none of our…

Tyler Tech is not a rock star outfit. They may think they are, but they are not.

I interviewed with them in 2008, and that remains, to date, the worst interview experience I have ever had. Everything they did before the interview seemed calculated to convince me to withdraw myself from consideration, and everything after seemed calculated to discourage anyone else I knew from applying.

So I felt a little frisson of schadenfreude from reading the article.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#94

Post by r00fus » Wed, Nov 30, 2016, 7:36 PM UTC

strictfp wrote: post #65

dkarl wrote: post #17

Similar problems have been reported in Tennessee and also in Indiana - where prosecutors have had a perhaps more troubling issue of inmates being mistakenly released early. It seems backwards to describe this as more troubling.

Better sorry than safe?

Where have we gone where this cornerstone of legal philosophy [1] is in question?

"It is better that ten guilty persons escape than that one innocent suffer"

[1] https://en.wikipedia.org/wiki/Blackstone's_formulation

Re: People’s freedom jeopardised by new software adopted by California’s courts

#95

Post by nkrisc » Wed, Nov 30, 2016, 7:52 PM UTC

panic wrote: post #6

Tyler Technologies acknowledged in a statement that the upgrade process had been “challenging” - but said poor training was to blame for bad inputting of data and integration with third-party applications that often introduce glitches into the system. People writing software need to take responsibility for how the software is used, especially when it can impact people's lives to this degree. You can't just blame thir…

I agree that you can't blame the software but disagree with who should be liable. The entity using the software needs to be liable (in this case either the State or the courts). They are responsible for implementing and using the the software thus ought to be liable for any mistakes it makes, whether user error or software bug. If they then want to take that up with the creator of the software, that's between them.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#96

Post by Domenic_S » Wed, Nov 30, 2016, 7:57 PM UTC

tener wrote: post #86

ThatGeoGuy wrote: post #60

Earlier quoted context omitted.

> Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA ( https://www.apega.ca ) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being abl…

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some cra…

> just churning out some crappy e-commerce website?

That can affect people's lives dramatically though. Off-by-1 error charges someone's debit card $2,000 instead of $200 and now they can't buy food. Ecomm is important.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#97

Post by noir_lord » Wed, Nov 30, 2016, 8:15 PM UTC

jdietrich wrote: post #81

treehau5 wrote: post #55

Earlier quoted context omitted.

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanis…

There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track dow…

Here in the UK we had a large NHS trust (regional sub department of our nationalised health service) shut down for 3 days, operations cancelled, delayed, surgeons stood down over malware.

http://www.computing.co.uk/ctg/news/2475950/lincolnshire-nhs...

Re: People’s freedom jeopardised by new software adopted by California’s courts

#98

Post by ThatGeoGuy » Wed, Nov 30, 2016, 8:32 PM UTC

tener wrote: post #86

ThatGeoGuy wrote: post #60

Earlier quoted context omitted.

> Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA ( https://www.apega.ca ) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being abl…

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.). But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some cra…

> But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some crappy e-commerce website?

Firstly engineering bodies are not a union. They are a legal organization that takes action to protect the public from actual harm. In the case of the software put forth in TFA, quantifiable harm was caused to people because of poorly tested and poorly integrated software.

Secondary to that, not every piece of code requires this level of oversight. As an example, you don't need to get a civil engineer to sign off so you can fix a picture frame on your wall; however, if you do decide to remove a load-bearing wall, you better be sure you need someone to sign that you're not collapsing your house, or selling it to someone else who won't know about the problem.

In the same way, the biggest restriction this creates for most people is that they won't be able to call themselves a software "engineer" and won't have the authority to sign off a project (in the same way you sign off that work on a bridge has been completed). This already exists in Canada and doesn't prevent anyone from creating a crappy e-commerce site. Protecting the word "engineer" is good for multiple reasons, and holding software to the same standards as we hold the rest of our public projects is not only a good idea in theory, but probably saves cost in the long run too. Think of all the people who will sue over being wrongfully arrested or put on the sex offenders registry.

> For any given piece of software there are many people responsible in various ways for its creation.

As with any large project. We don't blame the individual electrician and make them legally culpable if the light sockets start a house fire. However, if the sockets are faulty and the lead engineer _knew_ such, (or did not do enough due diligence such that they could not expect it to happen) yet still signed to let the construction finish and endangered someone who purchased the home, then the engineer is liable (to an extent for damages but more importantly that they will lose the right to work as an engineer in the future). The most salient point is that you create a code of responsibilities for engineers towards the public, and you build the legal framework so that projects that harm or endanger the public can be reigned in.

For projects like TFA, I can imagine that under an organization like APEGA, you could treat software similar to traditional projects by adding integration requirements to the contract, and requiring a certain period (90 - 120 days maybe?) of time after all the data has been transferred over to soft-test the system in production to see if it meets the on-site requirements (i.e. matches what is done manually). Does this make the contract more expensive? Yes. By having processes like these, can we be reasonably sure that we aren't going to cause a large amount of external harm or grief? Maybe. Building a culture where we don't just ship a proof-of-concept that can potentially imprison people because our manager had short term goals starts by making it clear who is liable, who is accountable, and where your responsibility lies.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#99

Post by nine_k » Wed, Nov 30, 2016, 8:48 PM UTC

dispose13432 wrote: post #84

nine_k wrote: post #80

Earlier quoted context omitted.

Compare it to automotive engineering. Cars are designed with an assumption that they will be sometimes mishandled, tipped over, crashed into things, run into bodies of water, set on fire, etc. In these circumstances cars should behave as to best preserve humans' safety, both inside (passengers) and outside (pedestrians). This leads to some serious design decisions on deeper level, e.g. making the engine go under the…

>Cars are designed with an assumption that they will be sometimes mishandled But not maliciously. If you run your car at another, the other car will break. I would compare it to lock-making. Despite thousands of years of lock-making, they _still_ get broken, and there's nothing to be done about it. >Deeply defensive programming methodologies also exist, from using safer languages and formally proven algorithms to pen…

Maliciously, too: see all these little things from door locks to built-in speed limiters.

Formal certification may be paperwork. But things like using e.g. Haskell instead of Ruby, or Rust instead of C, generating exhaustive tests where possible, making sure that tests touch every line of the code base, writing short pure functions for the most part, and isolating effectful and unsafe code are not about paperwork, to my mind.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#100

Post by treehau5 » Wed, Nov 30, 2016, 9:02 PM UTC

jdietrich wrote: post #81

treehau5 wrote: post #55

Earlier quoted context omitted.

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanis…

There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk. Some semi-hypothetical scenarios: A data leak in an appointment management app allows an abusive ex-partner to track dow…

I could go on ad nauseum about all the potential non-software ways that could go a long way in helping someone intent on breaking the law break the law, but the common thread in all your hypotheticals is people who want to break the law will find ways to break the law. If you want to help fix our current societal issues that drive people to want to break the law, that's another story.

We have armed guards, large vaults, and cameras in banking centers, but that doesn't stop people from trying to rob the bank.

We have federal laws and regulations protecting people's mail, but that doesn't stop identity thieves from going into your mailbox or digging through your trash for banking statements.

If a creepy stalker spots their ex's car driving down the road, what's stopping them from following her/him home? "But how would they find out what car they are driving?" you may rebute, well who decided to post what brand new car they drive on facebook? At some point, people have to take responsibility for the information they put out there about themselves. I wouldn't call into a radio show and blurt out my full name and address for everyone to hear before I gave my opinion.

I think this leads to a different issue -- why is appointment management software asking for personally identifiable information? Maybe we could make a distinction -- if you deal with PII, then these standards should be placed upon you.