Sounds like robopsychologist might become a real job after all.
For AI to thrive, it must explain itself
41–50 of 72 posts
Re: For AI to thrive, it must explain itself
#42Some people are missing the point. > "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law." It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates. > "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful…
Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that."
How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system.
If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.
Re: For AI to thrive, it must explain itself
#43Earlier quoted context omitted.
If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
Very much not. Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence. The software has no more agency than a cruise control.
Good luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.
Re: For AI to thrive, it must explain itself
#44Earlier quoted context omitted.
If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.
What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.
Re: For AI to thrive, it must explain itself
#45I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…
>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system. Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so." Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.
The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.
Re: For AI to thrive, it must explain itself
#46Just as with self-driving cars, you have to look at the counterfactual. We don't have an interpretable audit trail for the decisions made by human employees either.
So under the status quo, what happens in the case of an aberrant or problematic decision and how does society cope with this lack of interpretability?
We try to piece together a plausible, retrospective narrative by looking at fact patterns and taking into account the education, experience, actions, explanations, and rationalizations of other humans. (Trusting these accounts is further complicated by human traits such as self-interest, emotion, and cognitive biases.)
There exist entire professional specialties devoted to this problem (litigation, internal investigations, police detectives, accident boards) who spend a lot of time on this. Even so, much of the time we still don't really know why exactly people made the decisions they did.
This inconvenient fact does not stop us from employing humans, and society has not collapsed under the weight of the liability issues.
Re: For AI to thrive, it must explain itself
#47Some people are missing the point. > "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law." It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates. > "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful…
AI models simply exhibit the behaviour of the data they are trained on. If you make a model that tries to copy an individual persons behaviours, is it the AI model that is at fault or the person it's modelling? Where should the liability lay?
the problem with that also is that humans have emergent higher level reasoning, as a trivial, super simplified example, imagine if you trained a fictional AI on policy towards slaves before the worldwide ban on slavery, what sort of things would it suggest, its highly unlikely "banning slavery" would be one of them.
Re: For AI to thrive, it must explain itself
#48Some people are missing the point. > "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law." It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates. > "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful…
Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that." How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers…
On the other hand, can you actually make a human not repeat given action? Or explain it?
Re: For AI to thrive, it must explain itself
#49I don't think AI needs to be able to explain itself in order to be trusted. Human beings, in general, cannot give arbitrarily deep and valid explanations for their actions, and yet, somehow, we manage to come to trust them.
Re: For AI to thrive, it must explain itself
#50Earlier quoted context omitted.
Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.
What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.
It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.