Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For AI to thrive, it must explain itself

economist.com

Post reply on HN
Print view View original

11–20 of 72 posts

Re: For AI to thrive, it must explain itself

#11

Post by ghaff »

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so."

Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

Re: For AI to thrive, it must explain itself

#12

Post by chrisfosterelli »

ghaff wrote: post #11
chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system. Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so." Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

Re: For AI to thrive, it must explain itself

#13

Post by mseebach »

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place.

Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.

Re: For AI to thrive, it must explain itself

#14

Post by [deleted] »

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

[deleted]

Re: For AI to thrive, it must explain itself

#15

Post by kurthr »

chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition.

Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?

Re: For AI to thrive, it must explain itself

#16

Post by throwaway2048 »

mseebach wrote: post #13
chrisfosterelli wrote: post #3

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…

There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place. Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your acces…

If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".

Re: For AI to thrive, it must explain itself

#17

Post by ghaff »

chrisfosterelli wrote: post #12
ghaff wrote: post #11

Earlier quoted context omitted.

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system. Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so." Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat.

If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.

Re: For AI to thrive, it must explain itself

#18

Post by hinkley »

rdlecler1 wrote: post #5

>This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does. This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our ha…

If removing all of the spurious circuits is “cutting the bullshit” then what is left over is the truth.

Distilling a problem to its essence. Isn’t that the hardest step in explaining it?

Re: For AI to thrive, it must explain itself

#19

Post by throwaway2048 »

kazinator wrote: post #9
rdlecler1 wrote: post #5

>This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does. This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our ha…

If those elements of the graph are "spurious", why not remove them from the very beginning? Or, should I say, not have them there?

Because we dont know which ones are spurious and the parent is pretty optimistic about the idea of being able to analyize which ones are/are not.
Post reply on HN