For AI to thrive, it must explain itself

Post by moseandre » Sun, Feb 18, 2018, 5:09 PM UTC

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

Post by hinkley » Sun, Feb 18, 2018, 5:14 PM UTC

jonbarker wrote: post #10

narrator wrote: post #6

It would be an interesting breakthrough if Alpha Go could explain its reasoning for various plays. Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?

Since alphago plays the game differently (optimizing on odds to win, even if by a little, instead of the human method which involves intuition about shapes), it may be that its explanation if it were programmed to provide one would not provide any human readable 'insight' which an expert could then take to improve his or her game. Imagine a simpler example, some optimization solution using multiple dimensions and lin…

Post by gmueckl » Sun, Feb 18, 2018, 5:31 PM UTC

moseandre wrote: post #21

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. I have to be able to debug anything I work with; debugging is necessary. >> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" It's important to look into why the stop sign was m…

Post by darepublic » Sun, Feb 18, 2018, 5:38 PM UTC

gmueckl wrote: post #25

moseandre wrote: post #21

Earlier quoted context omitted.

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. I have to be able to debug anything I work with; debugging is necessary. >> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" It's important to look into why the stop sign was m…

I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).

Post by freeone3000 » Sun, Feb 18, 2018, 6:04 PM UTC

natch wrote: post #28

I’d like to simply know “how good is AI right now” at any given time. This information seems largely hidden. For example, can any AI expert tell me: what is the exact date when Target corporation crossed the threshold of being able to face match 20% of its customers across different stores with 20% accuracy? How about 50% (for both of the different numbers, to keep it simple). How about 80%? 90%? That’s just one exam…