Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are we using black box models in AI when we don’t need to? (2019)

hdsr.mitpress.mit.edu

Post reply on HN
Print view View original

71–80 of 200 posts

Re: Why are we using black box models in AI when we don’t need to? (2019)

#71

Post by taneq »

_jal wrote: post #29
taneq wrote: post #14

Earlier quoted context omitted.

It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint. Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.

Bank managers, however, can at least theoretically be hauled in to court and questioned.

That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#72

Post by MaupitiBlue »

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Isn’t the first sentence just an example of the second?

Re: Why are we using black box models in AI when we don’t need to? (2019)

#73

Post by rdlecler1 »

For god sakes. Enough of this black-box—we don’t understand how this works narrative. Neural networks are JUST computational circuits. Unfortunately there’s been a lot of laziness in cleaning those circuits up (W_ijk=O) to better reveal the topological circuitry.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#74

Post by xelxebar »

mkolodny wrote: post #39
rubyn00bie wrote: post #28

Earlier quoted context omitted.

I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample? I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much. Sorry, I'm just real confused by your…

Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself. Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your i…

It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors.

OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels.

Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion.

It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#75

Post by hyperpallium »

It's true our own cognition is also a black box, e.g. how do we recognize? It's as much a mystery to introspection as digestion. But the goal of science is to understand. Whereas with DL, we're just chimps in the dirt playing with sticks.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#76

Post by darkerside »

fhennig wrote: post #70
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> because despite being less accurate, they can have designers who can be held legally accountable and liable Are you implying you cannot be held accountable for black box models?

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

Re: Why are we using black box models in AI when we don’t need to? (2019)

#77

Post by MaxBarraclough »

MaupitiBlue wrote: post #72
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction. Isn’t the first sentence just an example of the second?

It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions.

That's not the same question as whether there were feelings involved, or whether the process was conscious.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#78

Post by throwawayjava »

mlthoughts2018 wrote: post #56
throwawayjava wrote: post #17

Earlier quoted context omitted.

I think it's a real open question whether the interpretable models are actually worse, or merely worse in competition/benchmark problem sets. The more deep models I build, the more I'm convinced that behind every inscrutable parameter hides a certain amount of overfitting, with maybe a few notable exceptions. E.g., can you build a decision tree that's not obviously overfit but is also susceptible to adversarial pertu…

It’s also a real open question whether any of the interpretable models are actually interpretable, or even if they are in any well-defined sense more interpretable than “black box” alternatives. In practice the answer is a massive “no” so far. Some of the least interpretable models I’ve had the misfortune to deal with in practice are misspecified linear regression models, especially when non-linearities in the true c…

> If your model does not adequately predict the data generating process, then your model absolutely does not explain it or articulate its inner working.

That's a very dynamicist viewpoint. I don't necessarily disagree.

However, in what sense to the prototypical deep learning models predict the data generating process?

I tend to agree that a lot of work with "interpretable" in the title is horseshit and misses the forest for the trees.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#79

Post by makomk »

jph00 wrote: post #53

This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example: > " For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depend…

No, they're not misrepresenting this at all. ProPublica's article https://www.propublica.org/article/machine-bias-risk-assessm... was pushing the claim that somehow, the COMPAS black box was implicitly deducing defendants' race from the "137 questions" input to it and labelling them likely reoffenders based on it in a way that was indepdendent of the key factors known to affect reoffending rates, such as age and gender. The paper in question seems to demonstrate the exact opposite is true: ProPublica were inadvertently using race as an imperfect proxy for age at sentencing, which is what the COMPAS algorithm really cared about, because their attempt to control for age at sentencing didn't work. After controlling for the actual age factor, COMPAS results didn't have any relationship to race anymore. (It's not even a weird weighting factor: predicted reoffending risk falls off rapidly with increasing age at first, then more slowly in a smooth fashion. It's just not linear.)

Now, it's of course possible to argue that judging reoffending risk based on age is in fact unfair and racist because it has disproportionate impact on certain racial groups, even though it's strongly predictive across all racial groups. That's not the argument ProPublica made, though. Their argument was about the supposed perils of black boxes, and they kind of acknowledged that age probably wasn't a racist criteria - or at least that it would be a lot harder to justify calling it one - by attempting to strip out its effects in the first place. It's also a different kind of argument entirely, one that revolves not around whether the algorithm is somehow treating people differently based on their detected race - because it isn't - but around what it means for a decision like this to be fair in the first place.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#80

Post by dchichkov »

1_over_n wrote: post #64
dchichkov wrote: post #6

I've happen to be in that workshop room and had chosen the 2% robot, not the 15% surgeon. If I remember correctly, the point of the question was determining willingness of society to use black box models, not challenging the need for models interpretability. Interpretability obviously doesn't hurt accuracy. But it is costly to engineer. And not always possible to make. Not always possible, because human capacity (and…

>> Interpretability obviously doesn't hurt accuracy. Why do you say this? From what i have seen it certainly can and does. For some industries finding a trade off where the magic is. https://link.springer.com/article/10.1007/s10664-018-9638-1 https://www.oreilly.com/ideas/predictive-modeling-striking-a...

You can take nearly any black box model and add some interpretability into it, without modifying this model. No modifications to the model, so accuracy stays the same.
Post reply on HN