71–80 of 171 posts
This is the first I've heard of this scandal. I'm curious what the major problems were with the software that caused the discrepancies. Dropped connections? Wikipedia didn't go into much detail with that. Ok I found this: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
Earlier quoted context omitted.
Your experience is quite limited. Too limited to have a meaningful opinion on this subject frankly.
Enlighten us then.
Hardware bugs are simply capsuled better so people dont recognize them as such. Its the same mechanism at play as for people who assume software normally functions. They are also just unaware of the inner working.
Stuff is just really complicated and people should manage their expectations. I get that going from assuming to have a solid base to no such thing existing is difficult to handle but there is no sensible way around that. Not doing so gets you stuff like the article.
After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.
> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat…
I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken.
Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts.
Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place.
Which incidentally is not entirely unlike the situation in this perversion of justice.
Every prosecutor in possession computer evidence of a crime having to prove the software correct is a nonstarter. Assuming it's right, and proving it buggy, is the right principle.
The standard does not have to be beyond all doubt. The standard is fine at beyond reasonable doubt. If the prosecution can replicate consistent evidence for and against the defendant using the same computer systems with simulated data, then that is probably enough to clear reasonable doubt.
But the burden of proof should absolutely always be on the party wishing to use "evidence" to take someone else's property, freedom, or life.
Assuming it's right is assuming guilt. That is the wrong principle.
As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
It also leads to the same horrible misjudgements as in the article mentioned above. If you dont know what to look for you will come to erroneous convictions by ruling out the error cases you do know.
After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.
As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
Your experience is quite limited. Too limited to have a meaningful opinion on this subject frankly.
I am having quite the big problem with this getting downvoted. The poster is spot on here and this needs to be communicated. Not doing so is irresponsible. Overconfidence in this regard is extremely dangerous and there is no nicer way to say this.
You can not reasonably assume that you are able to judge something works reliably without understanding how the error cases look. Especially if these are capsuled from you and or you dont know what to look out for.
This is at the very core of why something like the post scandal was possible. People repeating the same mistake in this very thread is just really bad. The only thing worse would be not telling them this.
The last time I looked into the Post Office scandal, the impression I got was that "we just had too much faith in the IT system!" was a convenient story, but the high-ups knew full well about the problems, and full well that they were persecuting innocent people, but did it anyway to cover up their own failings. If so, people need to see substantial jail terms for this. Perverting the course of justice is no misdemea…
When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they doubled down because they lack any sense of morality.
So while it is a sad technical tale it is also an indictment of British Board room culture.
The whole sorry affair has been covered by Private Eye magazine for over ten years.
Earlier quoted context omitted.
> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat…
Any sufficiently complex system operates constantly in failure mode
We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
Earlier quoted context omitted.
But thinking about a court case, what should be the presumption? Should it be up to the prosecution to prove that the RAID controller / network card didn't corrupt the data or that the GPU did the math correctly? Or should it be up to the defense to prove that it didn't? And how would each side demonstrate their case to a reasonable level of certainty? As others in this thread have said, for bits of the system that h…
Mathematical verification of both hardware and software should be mandatory, at least as long you rely on it for the law. Simple as that. For example, how can you trust a photo, when tons of computational photography has been applied to it, and you don't know the exact amount the algorithms can change the photo?
It does not seem very practical to throw out all digital evidence (eg photos and videos) because there is some chance of misprocessing. Who even records in analog today? What corresponding standard would you demand for analog evidence? Even in analog there were film issues, processing issues, noise, smudges, tampering and uncertainty, people have been dealing with these sorts of confounders for a long time.
We don't generally require "mathematical certainty" for other types of evidence. Personally I think GP was proposing a reasonably balanced commonsense perspective.