The presumption that computers are working correctly • Hacker News Board

Re: The presumption that computers are working correctly

#81

Post by psychoslave » Sun, Oct 01, 2023, 10:34 AM UTC

zubairq wrote: post #41

bmitc wrote: post #40

Earlier quoted context omitted.

> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat…

That makes sense, yes we have trained ourselves to see things that don’t work as working , like frogs in boiling water!

While the metaphore is great, it's actually stand on a myth.

That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits.

https://en.m.wikipedia.org/wiki/Boiling_frog

Re: The presumption that computers are working correctly

#82

Post by morelisp » Sun, Oct 01, 2023, 10:35 AM UTC

User23 wrote: post #16

topspin wrote: post #6

"Code is law." -Lawrence Lessig The UK apparently made that explicit. Doubtless others have as well: if we could forego the flag waving and whataboutery, That Would Be Great (tm).

Lawrence Lessig's greatest claim to fame is handing Disney and friends unlimited copyright before the Supreme Court. I'd hold up Judge Alsup if I wanted to give an example of a tech saavy legal practitioner.

While it's increasingly clear that Lessig's legacy will be as (to borrow a phrase) the internet's own failson, he is far more famous today for CC than Eldred v. Ashcroft.

Re: The presumption that computers are working correctly

#83

Post by switch007 » Sun, Oct 01, 2023, 10:42 AM UTC

dazzawazza wrote: post #78

didntcheck wrote: post #56

The last time I looked into the Post Office scandal, the impression I got was that "we just had too much faith in the IT system!" was a convenient story, but the high-ups knew full well about the problems, and full well that they were persecuting innocent people, but did it anyway to cover up their own failings. If so, people need to see substantial jail terms for this. Perverting the course of justice is no misdemea…

There is substantial evidence that the people in power in the Post Office Board dismissed the evidence from the sub post masters because they were, more often than not of Asian descent (Indian, Bangladeshi, Pakistani,...). So the Post Office Board had a mental block to joining the dots because they were racist. When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they d…

Private Eye is an absolute gem. It’s quite quirky and jokey but it’s the real deal if you want to know how this country is really run.

Re: The presumption that computers are working correctly

#84

Post by PartiallyTyped » Sun, Oct 01, 2023, 10:49 AM UTC

bmitc wrote: post #40

zubairq wrote: post #26

After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.

> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat…

> How can it possibly all work?

What must not to fail has very rigorous standards, e.g. Nasa, aerospace, and medical devices.

Everything is built like an onion of failsafes.

Re: The presumption that computers are working correctly

#85

Post by PartiallyTyped » Sun, Oct 01, 2023, 10:51 AM UTC

m-ee wrote: post #33

murderfs wrote: post #20

Earlier quoted context omitted.

As a programmer that works on drivers, it's my experience that hardware almost never works correctly. "We'll fix it in software" might as well be the official motto of the hardware industry.

As a firmware engineer I’m horrified at the presumption that hardware is always operating correctly. Many times I’ve been told “your new firmware is broken” and the real problem is something as mundane as an improper capacitor or resistor value. Also just look at the errata sheet for any microcontroller. Hardware bugs are so prevalent we have a special term and semi standardized document to describe them.

A good friend of mine is a hypervisor/kernel engineer, and I can swear he has said the same thing, almost word for word.

Re: The presumption that computers are working correctly

#86

Post by PartiallyTyped » Sun, Oct 01, 2023, 10:53 AM UTC

auggierose wrote: post #68

gwd wrote: post #48

Earlier quoted context omitted.

But thinking about a court case, what should be the presumption? Should it be up to the prosecution to prove that the RAID controller / network card didn't corrupt the data or that the GPU did the math correctly? Or should it be up to the defense to prove that it didn't? And how would each side demonstrate their case to a reasonable level of certainty? As others in this thread have said, for bits of the system that h…

Mathematical verification of both hardware and software should be mandatory, at least as long you rely on it for the law. Simple as that. For example, how can you trust a photo, when tons of computational photography has been applied to it, and you don't know the exact amount the algorithms can change the photo?

That's easy to say, until you've tried to write a provably correct piece of software or tried to do model checking.

I have tried to build models for the schemas that databases have, not their implementation, just the schema, and it is difficult AF. Using z3 for this might as well be trying to start a fire with a pair of rocks.

Re: The presumption that computers are working correctly

#87

Post by vjk800 » Sun, Oct 01, 2023, 10:53 AM UTC

The presumption becomes problematic mostly when the operator of the computer is the same party who is being persecuted. 1) It is difficult or impossible for any outsider to inspect the workings of the computer, 2) the operator has every incentive to fake the outputs of the computer or hide the errors.

Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs of the bittorrent swarm as an evidence in courts. Faking of this data would have been both very easy and pretty much impossible to detect by an outsider. Also they had a massive financial incentive to do so. The fact that this "evidence" was considered in the courts seemed outrageous to me.

Re: The presumption that computers are working correctly

#88

Post by cf141q5325 » Sun, Oct 01, 2023, 11:01 AM UTC

gwd wrote: post #48

murderfs wrote: post #20

Earlier quoted context omitted.

As a programmer that works on drivers, it's my experience that hardware almost never works correctly. "We'll fix it in software" might as well be the official motto of the hardware industry.

But thinking about a court case, what should be the presumption? Should it be up to the prosecution to prove that the RAID controller / network card didn't corrupt the data or that the GPU did the math correctly? Or should it be up to the defense to prove that it didn't? And how would each side demonstrate their case to a reasonable level of certainty? As others in this thread have said, for bits of the system that h…

You are treating hardware like software by arguing with the experience we have made so far with a given model. Environmental conditions as well as it being different physical objects is a huge problem for hardware. The data about error cases is also simply not recorded. You get at best statistical reliability with small sounding chances for errors that are still happening all the time.

You can also not make any statement about reality by arguing with feasibility.

edit: I am unfortunately not qualified to tell you how feasible ways to deal with reality in this case could look. Which doesnt mean that i cant identify some error cases. You also require different levels of competence for designing a rocket and recognizing a non functioning one.

edit1: In regards to the usage of statistics as a basis one should also remember confidence and these cases https://www.science.org/content/article/unlucky-numbers-figh...

Re: The presumption that computers are working correctly

#89

Post by agentgumshoe » Sun, Oct 01, 2023, 11:03 AM UTC

vjk800 wrote: post #87

The presumption becomes problematic mostly when the operator of the computer is the same party who is being persecuted. 1) It is difficult or impossible for any outsider to inspect the workings of the computer, 2) the operator has every incentive to fake the outputs of the computer or hide the errors. Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs…

Ah the Prenda Law saga, such memories

Re: The presumption that computers are working correctly

#90

Post by harry8 » Sun, Oct 01, 2023, 11:03 AM UTC

murderfs wrote: post #20

phire wrote: post #7

As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.

As a programmer that works on drivers, it's my experience that hardware almost never works correctly. "We'll fix it in software" might as well be the official motto of the hardware industry.

What you say is completely true in my experience. Comically so.

However when using a computer application and noticing unexpected and erroneous behaviour, what proportion is the time can the bug be traced back to hardware vs software?

I’m going with about 1 to 1000. I think that’s conservative. Do you see it differently?