Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are we using black box models in AI when we don’t need to? (2019)

hdsr.mitpress.mit.edu

Post reply on HN
Print view View original

101–110 of 200 posts

Re: Why are we using black box models in AI when we don’t need to? (2019)

#101

Post by rwallace »

daoxid wrote: post #89
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

> even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers. I put it to you that having questions answered with plausible fabrications, is actually a significantly worse situation than not yet being able to ask the questions at all. At least in the latter situation, we know what we need to be working on.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#102

Post by hooande »

1e-9 wrote: post #38

The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you shoul…

This isn't how it works, at all. You're going to get better results with 1,000 parameters than with 100 explainable ones. There's a limit to how much humans can understand. We use machine learning to surpass that limit

Re: Why are we using black box models in AI when we don’t need to? (2019)

#103

Post by hooande »

I don't know how many of you are familiar with Kaggle competitions, but the winning scores on the leaderboards can come down to 0.0001 points of accuracy or less. Winning something like that with an explainable model is like having one hand tied behind your back.

If tiny fractions of a point matter (the rare case), then making the model explainable is adding a ton of complexity for little practical gain. It sounds nice, but the numbers are unforgiving

Re: Why are we using black box models in AI when we don’t need to? (2019)

#104

Post by chasedehan »

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?

Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc)

Not sure what point you are trying to make here.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#105

Post by daoxid »

knzhou wrote: post #92
daoxid wrote: post #89

Earlier quoted context omitted.

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to stat…

> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.)

> You can already do that. Just change that number in the input and see how the output changes.

Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#106

Post by hooande »

rubyn00bie wrote: post #44

> [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer. If your data is off, your model might be in…

The goal of science is to produce a testable hypothesis. The world is full of things that people can 'explain', or even prove with mathematics, but don't hold up to basic experimental testing or reproducibility.

Every new theory has a plain english explanation that's easy to understand. Few of them have the raw accuracy or reliability of top ML models

https://en.wikipedia.org/wiki/Replication_crisis

Re: Why are we using black box models in AI when we don’t need to? (2019)

#107

Post by daoxid »

dcposch wrote: post #93
daoxid wrote: post #89

Earlier quoted context omitted.

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

> we can ask them questions in order to find out how they came to a particular conclusion. For the most basic cognitive tasks, we typically can't. If you show someone a picture of a cat and a dog, they can easily recognize which is which. If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#108

Post by gusostow »

chasedehan wrote: post #104
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations? Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc) Not sure what point you are trying to make here.

Those models are ensembles of decision trees. A single tree is interpretable, but sensitive to overfitting.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#109

Post by Mathnerd314 »

jph00 wrote: post #51
RSchaeffer wrote: post #16

Earlier quoted context omitted.

But how did their model compare against others? The article only mentions how their interpretable model compared against their own ML attempts

Their model didn't win. IBM's model won, based on actual metrics around useful insights.

The IBM team got $5,000 and the second place/honorable mention NYU got $2,000. So going by prize amounts, the Duke model was still pretty good.

IBM turned the model/paper into a toolkit: https://www.ibm.com/blogs/research/2019/08/ai-explainability... Their model seems to be a variant of decision trees that has a knob controlling how complicated the trees are.

And the evaluation was completely subjective, so there's not any meaning to the Duke people losing besides that the judges didn't like them.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#110

Post by lonelappde »

daoxid wrote: post #107
dcposch wrote: post #93

Earlier quoted context omitted.

> we can ask them questions in order to find out how they came to a particular conclusion. For the most basic cognitive tasks, we typically can't. If you show someone a picture of a cat and a dog, they can easily recognize which is which. If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.

Introspectable Higher level reasoning is irrelevant. Any introspectable higher level reasoning a person can do has already been automated for efficiency.
Post reply on HN