Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are we using black box models in AI when we don’t need to? (2019)

hdsr.mitpress.mit.edu

Post reply on HN
Print view View original

51–60 of 200 posts

Re: Why are we using black box models in AI when we don’t need to? (2019)

#51

Post by jph00 »

RSchaeffer wrote: post #16
mkl wrote: post #13

They never got to the results of the competition and how the interpretable model did! From https://www.fico.com/en/newsroom/fico-announces-winners-of-i... : "The team representing Duke University, which included Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang and Tong Wang, received the FICO Recognition Award acknowledging their submission for going above and beyond expectations with a fully…

But how did their model compare against others? The article only mentions how their interpretable model compared against their own ML attempts

Their model didn't win. IBM's model won, based on actual metrics around useful insights.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#52

Post by 1_over_n »

StuffedParrot wrote: post #47
rubyn00bie wrote: post #44

> [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer. If your data is off, your model might be in…

> So uhhh, isn't this like not science? Very little of technology has anything to do with validating hypotheses. > meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model. This is…

At our company, we are working with "mechanistic" or mathmatical models rather than stastical approches. I have observed :

1. It is hard than it should be to explain the concept to people (particularly VCs) 2 . people struggle to understand that a mechanistic model could have more utlity than a machine learning black box 3. people think you are doing something wrong if you are not using a neural network 4. The less people understand about neural networks, the more they seem to believe they are appropriate for all predictive / modelling problems 5. There is generally quite a low understanding of scientific method in the startup / VC space (speaking as someone who has worked in and around academia for years) vs how "scientific" people believe they are because it sounds good to be data driven and scientific about running startups and funding them.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#53

Post by jph00 »

This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example:

> "For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depended on race, age, and criminal history, that COMPAS itself must depend on race. However, when one approximates COMPAS using a nonlinear model, the explicit dependence on race vanishes, leaving dependence on race only through age and criminal history. This is an example of how an incorrect explanation of a black box can spiral out of control."

The concern about the strong relationship between race and COMPAS predictions is not largely based on a concern about whether there is an explicit dependence in the model. The concern is whether there's a relationship either explicitly or implicitly. And in particular, whether such a relationship results in unfair outcomes. The findings of the ProPublica study (https://www.propublica.org/article/how-we-analyzed-the-compa...) strongly suggested this was the case:

"- Black defendants were often predicted to be at a higher risk of recidivism than they actually were. Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).

- White defendants were often predicted to be less risky than they were. Our analysis found that white defendants who re-offended within the next two years were mistakenly labeled low risk almost twice as often as black re-offenders (48 percent vs. 28 percent).

- The analysis also showed that even when controlling for prior crimes, future recidivism, age, and gender, black defendants were 45 percent more likely to be assigned higher risk scores than white defendants."

I understand the desire of the MIT researchers to promote the value of their work, but in this case they appear to be doing so in a potentially damaging way.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#54

Post by 1_over_n »

1e-9 wrote: post #38

The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you shoul…

I would also add in that blackbox ML solutions are deceptively "simple" approaches. Like a miraj in the desert for software developers: No domain expertise neccessary - or even the counter narriatives may come in... "domain expertise holds you back - you need to be an outsider with a new method", where as many problems require domain expertise for a realistic understanding of the complexity of the problem-solution.

Other narriatives that can creep in "once we get enough data..." which for some problems may as well be never.

Also, for some systems pertubing the system changes the problem meaning your dataset and model may not reflect changes in a dyanmic / complex system.

Also, blackbox ML approaches struggle to combine data from multiple modalities which is often relevant to many real world problems (at least with the majority of algorithms which are realistically implementable off the shelf).

Re: Why are we using black box models in AI when we don’t need to? (2019)

#55

Post by baddox »

rubyn00bie wrote: post #44

> [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer. If your data is off, your model might be in…

> your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science

But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.

Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.

But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#56

Post by mlthoughts2018 »

throwawayjava wrote: post #17
owenshen24 wrote: post #5

My understanding is that interpretable models, especially for neural networks, are very far and away from state of the art performance. Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models [1]. Even in computer vision, which is where I think they've been most successful, the visualization techniques used seem more suggestive, then explanatory [2]. [1] http://…

I think it's a real open question whether the interpretable models are actually worse, or merely worse in competition/benchmark problem sets. The more deep models I build, the more I'm convinced that behind every inscrutable parameter hides a certain amount of overfitting, with maybe a few notable exceptions. E.g., can you build a decision tree that's not obviously overfit but is also susceptible to adversarial pertu…

It’s also a real open question whether any of the interpretable models are actually interpretable, or even if they are in any well-defined sense more interpretable than “black box” alternatives.

In practice the answer is a massive “no” so far. Some of the least interpretable models I’ve had the misfortune to deal with in practice are misspecified linear regression models, especially when non-linearities in the true covariate relationships causes linear models to give wildly misleading statistical significance outputs and classical model fitting leads to estimating coefficients of the wrong sign.

Real interpretability is not a property of the mechanism of the model, but rather consistent understanding of the data generating process.

Unfortunately, people like to conflate the mechanism of the model for some notion of “explainability” because it’s politically convenient and susceptible to arguments from authority (if you control the subjective standards of “explainability”).

If your model does not adequately predict the data generating process, then your model absolutely does not explain it or articulate its inner working.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#57

Post by andlima »

rubyn00bie wrote: post #44

> [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer. If your data is off, your model might be in…

Machine Learning is about solving a task by automatically learning patterns based on examples and trying to generalize to new data. Granted, it's not necessarily the best approach to understand deeply how a phenomenon occurs.

You could use a black box model if you're more interested in predicting correctly images of handwritten digits than in understanding how the pixels relate to each other.

Of course, usually people want both accuracy and interpretability. It boils down to understanding what's more important for the problem at hand and making the compromises accordingly.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#58

Post by OrgNet »

Gaelan wrote: post #37
OrgNet wrote: post #22

why include 2019 in the title of this post? was it automatically added (is that a bug)?

There's a (human-enforced) HN rule to do that for old articles, so they're not mistaken for news. I'm not sure what the standard for "old" is, but it isn't 2019 anymore, so I guess it's technically correct.

2 months old is usually not considered old news on this site... I think that their algo is buggy on year-change

Re: Why are we using black box models in AI when we don’t need to? (2019)

#59

Post by Gaelan »

OrgNet wrote: post #58
Gaelan wrote: post #37

Earlier quoted context omitted.

There's a (human-enforced) HN rule to do that for old articles, so they're not mistaken for news. I'm not sure what the standard for "old" is, but it isn't 2019 anymore, so I guess it's technically correct.

2 months old is usually not considered old news on this site... I think that their algo is buggy on year-change

As far as I know, there's no algorithm doing it. People put it in the title themselves, or mods do if they forgot.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#60

Post by Rapzid »

m0zg wrote: post #40

Human cognition itself is a black box model. We routinely make snap decisions first and then justify them after the fact. Anything that requires serious consideration is darn near insurmountable for a human mind in the presence of incomplete information, emotions, and conflicting motivations. So while I'd like neural nets to be more interpretable, to me it'd take a distant second place. The first would be to get mode…

I share this sentiment.

It occurs to me that human decisions making can be either explained a very high level, or a very low level(neurons firing). But the magic in between is too complicated to draw direct lines between stimuli and decisions.

AI seems to be the same way. We have statistical models that explain how most everything works at a low level. That a simple math formula can approximate just about anything. How we arrive at weights that separate data sets in different dimensions and etc. For convolution networks we also understand how layer "decisions" flow through to other layers. But, it's too complicated to look at an image and explain exactly how the input pixels will result in a classification output.

I'm not sure how much I should care about that.. Seems like a problem for mathematicians/statisticians. Having the doctor explain himself gives me no more true insight into how his brain works than having somebody explain at a high level how the robot was trained and reacts in different situations.

Post reply on HN