Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

People’s freedom jeopardised by new software adopted by California’s courts

bbc.co.uk

Post reply on HN
Print view View original

81–90 of 131 posts

Re: People’s freedom jeopardised by new software adopted by California’s courts

#81

Post by jdietrich »

treehau5 wrote: post #55
jdietrich wrote: post #26

Earlier quoted context omitted.

We need formal codes of practice, we need good institutions, but we also need a cultural change. We no longer have the luxury of saying "oh, it's just a CRUD app, it doesn't matter if things go wrong". Software has become too important, it has become too deeply intertwined in our daily lives. Errors and leaks from trivial little apps can have life-changing consequences for users. If your software handles personally i…

Meh. That's all I have to say about that. We already have the distinction between software that can actually affect people's lives in a negative/positive way and software that is merely a nuance or inconvenient. The former is heavily regulated. Mistakes in financial accounting software, healthcare software, or mission critical software such as industrial automation or say launching satellites cause companies to vanis…

There's no such thing as harmless software any more. Our devices are too interconnected and we are too reliant upon them. Failure modes can cascade in unpredictable ways. "Harmless" systems often contain potentially harmful data; highly sensitive systems often share network resources with IoT junk.

Some semi-hypothetical scenarios:

A data leak in an appointment management app allows an abusive ex-partner to track down and kill their victim.

Mobile malware spread by an in-game advertising network causes a massive DDoS of the cellular network. Some areas are left without cell coverage for several hours while operators scramble to reconfigure their base station equipment; thousands of 911 calls go unanswered until service is resumed.

Poor encryption in a chat app allows an oppressive regime to identify, surveil and "disappear" human rights activists.

A bug in an integrated logistics management system causes some parcels to be tagged with the wrong attributes. A consignment of temperature-sensitive medicine is inadvertently shipped in an unrefrigerated container; the mistake goes unnoticed until patients start dying, because the "smart" temperature sensor inside the package thought that the shipment was safe to transport at ambient temperatures.

A disreputable dating app is hacked and the names of the users published, leading to several suicides.

A wifi-enabled printer is connected to the secure side of a hospital network by an unwitting member of staff. The printer is pwned, bringing down the entire network for 36 hours. Two ICU patients die as a result of preventable errors, because their medical records were unavailable.

Several models of aftermarket in-car entertainment units are pwned due to an unpatched vulnerability in Android. The units are connected to the CAN-bus in order to use the media controls on the steering wheel. The poorly-written malware intermittently floods the CAN-bus with malformed packets, causing sudden failures of the braking system in several cars.

I could go on ad nauseum.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#82

Post by dsfyu404ed »

ptaipale wrote: post #61
andrewla wrote: post #36

Earlier quoted context omitted.

It definitely seems like more of the blame here should be shouldered by the police and the courts, who, knowing that the system has problems (and hopefully assured that those problems will be resolved in the future) should put less faith in them, and double-check (against filed papers, for example) potentially dubious results. This is more expensive in terms of people's times, but it's just part of the cost of adopti…

Indeed. The "Computer says no" or "Computer says catch him" attitude should change: particularly if you know it's a new system, check twice before making drastic actions based on data it gives.

Even just "computer says this record has been randomly selected for manual review to verify system integrity, please consult your administrator on how to proceed or enter a manager's PIN to override" would be really helpful since it would effectively train people in double checking.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#83

Post by heisenbit »

Terr_ wrote: post #77
jdietrich wrote: post #58

Earlier quoted context omitted.

>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to…

> Nuremberg defence Except we're not just talking about clearly malicious or fraudulent software with sins of commission. The bigger issue is cutting corners, taking shortcuts, and individual weaknesses that break the whole system.

Successive and cumulative corner cutting. And management should provide oversight covering several steps. But often they close their eyes and the last developer who touched the code is getting the blame.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#84

Post by dispose13432 »

nine_k wrote: post #80
dispose13432 wrote: post #70

Earlier quoted context omitted.

The threat model of code is different than that of bridges, and legislation won't help. Civil engineers have to make sure that bridges won't naturally collapse. Software "engineers" have to ensure that no one will break their code, from the privacy and anonymity of their house, with millions of dollars in payoffs. If bridges held millions of dollars and would be accessible from around the world, they would be breakin…

Compare it to automotive engineering. Cars are designed with an assumption that they will be sometimes mishandled, tipped over, crashed into things, run into bodies of water, set on fire, etc. In these circumstances cars should behave as to best preserve humans' safety, both inside (passengers) and outside (pedestrians). This leads to some serious design decisions on deeper level, e.g. making the engine go under the…

>Cars are designed with an assumption that they will be sometimes mishandled

But not maliciously. If you run your car at another, the other car will break.

I would compare it to lock-making. Despite thousands of years of lock-making, they _still_ get broken, and there's nothing to be done about it.

>Deeply defensive programming methodologies also exist, from using safer languages and formally proven algorithms to pen-testing the completed setup. Their adoption just costs effort and time, that is, money.

It's not just money. It's paperwork, bureaucracy, and ultimately not effective. Practically, the only way to ensure safety is to take down the internet.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#85

Post by tomjen3 »

CodeMage wrote: post #35
tener wrote: post #32

Earlier quoted context omitted.

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they…

I would amend that to "don't put the blame solely on the developers". I've seen way too many developers whose mentality is "ship it, it seems good enough, we'll fix any bugs as we go".

Seems perfectly reasonable to me.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#86

Post by tener »

ThatGeoGuy wrote: post #60
tener wrote: post #32

Earlier quoted context omitted.

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they…

> Too frequently they are put under huge pressure and cannot say "no" to their managers. Actually, this is precisely why something like the "Iron Ring", linked above, exists. Standards bodies like APEGA ( https://www.apega.ca ) exist precisely so that you have someone backing you if you say "no" to a manager. If you're an engineer and you put the public at harm in any way, APEGA will come after you. Part of being abl…

Software projects that have significant impact on public safety/well being are a relatively rare kind. You can easily apply additional policies on them - and indeed this is what is being done (see airplane software, medical software etc.).

But will you really call your union because your manager forces you to skip writing unit tests for each and every class you write? Having in mind you are just churning out some crappy e-commerce website?

Completely another thing is that you might be careful with your code, the third party components you are using are more lax. This is well recognized e.g. in medical software where any third party code must identified and held to the same standards as the software you write.

It is very difficult to say who is to blame for creating a certain software bugs. Is it the person that configures the software, becuase he chose invalid config parameters? Is it the intern that wrote invalid piece of parsing code? Is it the system architect who put wrong data into the specification?

For any given piece of software there are many people responsible in various ways for its creation. In such contexts you cannot be blaming individual persons especially that their intentions might be good. Even very careful programmers will make bugs.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#87

Post by hackuser »

jdietrich wrote: post #58
tener wrote: post #32

Earlier quoted context omitted.

Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. There are people which are responsible for delivery, testing and they are paid way more than the average developer. Unlike in engineering, with software you can - and indeed should - test it before deployment in thousands of ways. Finally there is also a receiving party that should ensure they…

>Don't put the blame on the developers. Too frequently they are put under huge pressure and cannot say "no" to their managers. Befehl ist Befehl. Blaming management is literally the Nuremberg defence. If we feel that we have to do reckless and dangerous things in order to keep our jobs, then we desperately need to unionise. It's just not good enough for us to throw up our hands and blame the PHBs. Management need to…

> Pressure needs to come from users, but also from developers

It needs to come from government especially. It's a basic function of government to protect the public

> How would we react if this story was about a plane crash or a nuclear accident caused by negligence?

If there were no government laws penalizing this outcome and regulations preventing it, I'd be horrified.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#88

Post by hackuser »

dispose13432 wrote: post #70
noir_lord wrote: post #19

Software is currently in a strange place legislatively, in the 18th century Civil Engineering exploded (sometimes literally) and the number of disasters went up in radically (for an interesting case https://en.wikipedia.org/wiki/Tay_Bridge_disaster ), over time professional standards bodies grew alongside the maturing industry to both ensure that people working in the industry where adequately trained and adequately…

The threat model of code is different than that of bridges, and legislation won't help. Civil engineers have to make sure that bridges won't naturally collapse. Software "engineers" have to ensure that no one will break their code, from the privacy and anonymity of their house, with millions of dollars in payoffs. If bridges held millions of dollars and would be accessible from around the world, they would be breakin…

Are you saying it's impossible to write quality code?

There is high-assurance software these days.

Re: People’s freedom jeopardised by new software adopted by California’s courts

#89

Post by joesmo »

Justice is whatever the 'justice' systems says it is. Citizens have no recourse. The 'justice' system has no checks or balances that can be accessed, except by the rich. There are no penalties for abuse or misuse, even when uncovered.

Yeah, that's exactly what I think of when I think of the concept of 'justice'. One day some people might create a just society, but it almost certainly won't be in the US.

Post reply on HN