Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are we using black box models in AI when we don’t need to? (2019)

hdsr.mitpress.mit.edu

Post reply on HN
Print view View original

81–90 of 200 posts

Re: Why are we using black box models in AI when we don’t need to? (2019)

#81

Post by perl4ever »

MaxBarraclough wrote: post #77
MaupitiBlue wrote: post #72

Earlier quoted context omitted.

> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction. Isn’t the first sentence just an example of the second?

It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions. That's not the same question as whether there were feelings involved, or whether the process was conscious.

"The point is that the explanation given does not correspond to the process through which the decision was truly reached."

Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, one should be cautious of assuming how it works even if you're not doing it by reflex. Psychological explanations are the rationalizations that people make after the fact.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#82

Post by _jal »

taneq wrote: post #71
_jal wrote: post #29

Earlier quoted context omitted.

Bank managers, however, can at least theoretically be hauled in to court and questioned.

That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.

Yes, I'm (somewhat) aware of the research. I'm unaware of any legal jurisdiction that recognizes it, let alone tries to address it somehow.

There's reality, there's legal reality, and then there's what legal reality does to reality.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#83

Post by devit »

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#84

Post by boltzmannbrain »

A couple salient quotes from "Interpreting AI Is More Than Black And White" in Forbes [1]:

"No matter the definition, developing an AI system to be interpretable is typically challenging and ambiguous. It is often the case that a model or algorithm is too complex to understand or describe because its purpose is to model a complex hypothesis or navigate a high-dimensional space, a catch-22. Not to mention what is interpretable in one application may be useless in another."

"Underspecified and misaligned notions of interpretation impede progress towards the rigorous development of understandable, transparent, trusted AI systems."

[1] https://www.forbes.com/sites/alexanderlavin/2019/06/17/beyon...

Re: Why are we using black box models in AI when we don’t need to? (2019)

#85

Post by sojournerc »

devit wrote: post #83
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.

But that explanation is likely full of cognitive bias, and all sorts of subjective experience.

Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#86

Post by jph00 »

makomk wrote: post #79
jph00 wrote: post #53

This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example: > " For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depend…

No, they're not misrepresenting this at all. ProPublica's article https://www.propublica.org/article/machine-bias-risk-assessm... was pushing the claim that somehow, the COMPAS black box was implicitly deducing defendants' race from the "137 questions" input to it and labelling them likely reoffenders based on it in a way that was indepdendent of the key factors known to affect reoffending rates, such as age and gend…

Whether a variable is latent or explicit isn't really relevant to the question of algorithmic fairness.

The link I provided gives the actual details of the method and findings; this is probably a more useful source for the details. The claim that the actual source of the difference is 'age' doesn't really make sense. There isn't enough of a difference in the number of young people between black and white populations to result in the differences found in the analysis.

(I do agree that the actual attempt to control for age was poorly done; it really shouldn't have been done at all, since it had nothing useful to add to the analysis or results.)

PS: It's 'COMPAS', not 'COMPASS'.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#87

Post by i000 »

While I can definitely see the value of black box models in engineering and commercial applications (including clinical ones), I am quite troubled by the new hype of applying to deep learning to so many basic scientific problems. The 'deep' models do not tend to result in deeper insights - which idealistically ought to be the goal.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#88

Post by shakna »

psv1 wrote: post #31
shakna wrote: post #26

Earlier quoted context omitted.

Shouldn't this fall into the "trust by verify" category of things? Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour. You have to trust it on new data. You can always verify why it made the choices it did on old data. That seems like something incredibly helpful…

> You can always verify why it made the choices it did on old data. This makes no sense. Your model typically isn't asked to make predictions on "old data" because it was trained on it. And understanding why it made certain prediction isn't any different on "new" vs old data. You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.

> You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.

Or, like in the article, you can make use of introspective models that can tell you what dimensions were fitted. This can more easily show you where overfitting is occurring.

You can see the weights of the model and follow the decision network, which allows you to explain why something has occurred in a much clearer fashion.

An introspective model gives you a much clearer picture of why certain choices were made, rather than the fuzzy picture you get running statistical analysis against a black-box'd algorithm.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#89

Post by daoxid »

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well.

Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

Re: Why are we using black box models in AI when we don’t need to? (2019)

#90

Post by somurzakov »

When I want to predict something, I need the highest accuracy, and I am not interested in interpretability of an individual parameter.

and interpretability is only a half measure. If i really want to understand the model, I will conduct causal analysis with randomized controlled trial and will find the root cause and impact of the factor of my interest.

Post reply on HN