For AI to thrive, it must explain itself
economist.com
For AI to thrive, it must explain itself
1–10 of 72 posts
Re: For AI to thrive, it must explain itself
#2Great article. I found a library that works with Keras last weekend; it shows which input features most strongly contribute to classifications. My little experiment: http://blog.markwatson.com/2018/02/trying-out-integrated-var...
Re: For AI to thrive, it must explain itself
#3I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.
In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interprets data from the components. If there was a crash, they can tie it down to "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" without needing much interpretability of the neural network directly. If it's an end-to-end system, such as the nvidia car which connects the camera inputs directly to a wheel output, then explainability of that network is much more important.
There was a poll I saw on twitter a while ago (can't seem to find now) that had asked if users would prefer to be treated by an AI doctor with low explainability but 99% accuracy, or high explainability but only 70% accuracy. I think context is key here as well. If the expert doctor working in tandem with the explainable system is better than the non-explainable system, you'd want the explainable system. But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.
Re: For AI to thrive, it must explain itself
#4'“the reason to give reasons is so that others will evaluate your actions and beliefs”. Today’s autonomous machines do not have their own interests to serve. Instead, their explanations are forged by and for human beings.'
This seems like the fundamental difference between AI and human intelligence at the moment. Our intelligence is based on our social prowess. We're always competing with, and exchanging ideas with our peers. AI should be modeled based on this.
Re: For AI to thrive, it must explain itself
#5>This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does.
This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our hands up. We can apply algorithms to trim out the spurious weights and what we’re left with is a logical circuit that we can analyze. Show an electrical engineer the circuit diagram of a 3 bit added and she’ll know exactly the function. Add a bunch of spurious circuits to that same diagram and of course it’s not going to make sense.
Re: For AI to thrive, it must explain itself
#6It would be an interesting breakthrough if Alpha Go could explain its reasoning for various plays.
Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?
Re: For AI to thrive, it must explain itself
#7[deleted]
Re: For AI to thrive, it must explain itself
#8I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…
> explainability is necessary to use it.
For that matter, freedom from daily blue screens and reboots is not needed to use an operating system.
All sorts of non-AI software has confusing states and quirks, yet is in wide use.
Re: For AI to thrive, it must explain itself
#9>This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does. This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our ha…
If those elements of the graph are "spurious", why not remove them from the very beginning? Or, should I say, not have them there?
Re: For AI to thrive, it must explain itself
#10It would be an interesting breakthrough if Alpha Go could explain its reasoning for various plays. Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?
Since alphago plays the game differently (optimizing on odds to win, even if by a little, instead of the human method which involves intuition about shapes), it may be that its explanation if it were programmed to provide one would not provide any human readable 'insight' which an expert could then take to improve his or her game. Imagine a simpler example, some optimization solution using multiple dimensions and linear algebra. A description of how this works would be unreadable to most people. Since most readable language operates in the world of at most four dimensions without introducing jargon, this is the real problem, not that the program cannot explain itself.