Why are we using black box models in AI when we don’t need to? (2019) • Hacker News Board

Re: Why are we using black box models in AI when we don’t need to? (2019)

#91

Post by danShumway » Sun, Jan 05, 2020, 1:32 AM UTC

darkerside wrote: post #76

fhennig wrote: post #70

Earlier quoted context omitted.

> because despite being less accurate, they can have designers who can be held legally accountable and liable Are you implying you cannot be held accountable for black box models?

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain.

AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for using the system. You can still hold manufacturers and businesses accountable for promising a degree of accuracy that their system couldn't meet.

In the same way, if my dog bites someone, and I get sued and argue, "animal motivation is really hard, we don't know why the dog chose to do that", I'm not going to win the case. The reason it happened is because I didn't leash my dog.

The reason the black box hurt someone is because an accountable human improperly used it or inaccurately advertised it.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#92

Post by knzhou » Sun, Jan 05, 2020, 1:35 AM UTC

daoxid wrote: post #89

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion.

No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

> So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#93

Post by dcposch » Sun, Jan 05, 2020, 1:39 AM UTC

daoxid wrote: post #89

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

> we can ask them questions in order to find out how they came to a particular conclusion.

For the most basic cognitive tasks, we typically can't.

If you show someone a picture of a cat and a dog, they can easily recognize which is which.

If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

Re: Why are we using black box models in AI when we don’t need to? (2019)

#94

Post by q_andrew » Sun, Jan 05, 2020, 1:45 AM UTC

daoxid wrote: post #89

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

This is actually false. Early psychologists thought that humans could know everything about their own brain's processes, but observational psychology proved most of their assumptions wrong. It's called introspection, and it's almost always not indicative of a person's actual inner workings.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#95

Post by jlarocco » Sun, Jan 05, 2020, 1:46 AM UTC

daoxid wrote: post #89

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…

Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?

Re: Why are we using black box models in AI when we don’t need to? (2019)

#96

Post by szczepano » Sun, Jan 05, 2020, 1:48 AM UTC

What a dumb question is it? The same logic is behind sports and nobel price awards.

It's easier to say to people who make financial decision that this is great artificial intelligence that will solve all your problems then explain mathematical equations behind it that 90% people won't understand. The more complicated the stuff is the higher probability it will be sold. That's the standard in IT world from quite some time.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#97

Post by owenshen24 » Sun, Jan 05, 2020, 1:56 AM UTC

chestervonwinch wrote: post #46

owenshen24 wrote: post #5

My understanding is that interpretable models, especially for neural networks, are very far and away from state of the art performance. Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models [1]. Even in computer vision, which is where I think they've been most successful, the visualization techniques used seem more suggestive, then explanatory [2]. [1] http://…

> Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models Well, that and a decision tree isn't any more interpretable if its nodes are all filled with `Pixel[i,j] > Threshold` -- or is the idea that you would somehow extract logical predicates by tracing paths through the tree or glean interpretation from those predicates?

I agree that large decision trees are also hard to interpret. The idea in the above paper is that we regularize with respect to average tree path length, based on the intuition that smaller trees are more interpretable. (But the issue about pixel thresholds probably still stands.)

Re: Why are we using black box models in AI when we don’t need to? (2019)

#98

Post by xmly » Sun, Jan 05, 2020, 2:00 AM UTC

danShumway wrote: post #91

darkerside wrote: post #76

Earlier quoted context omitted.

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain. AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for usi…

There are payment systems based on human facial recognition. If the model failed, it means losing money. Apparently you can not deny liability because of using a black-box model.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#99

Post by bitL » Sun, Jan 05, 2020, 2:02 AM UTC

I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of their interpretable models, then acting as arbiters and judges of the work of others, preventing future advancements of the field.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#100

Post by autokad » Sun, Jan 05, 2020, 2:10 AM UTC

1e-9 wrote: post #38

The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you shoul…

> The main advantage of a blackbox ML solution is shorter development time to a useful performance level.

I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16 GPU instances over a couple weeks of training )

meanwhile, I consider gradient boosting fairly interpretable and they can get pretty close results with a lot less tweaking and training time. If you want to go full non-black box, logistic regression with l1 penalization and only a little bit of feature engineering often does really well, probably a lot less development time / cost compared to those high cost PHD research scientists.