Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are we using black box models in AI when we don’t need to? (2019)

hdsr.mitpress.mit.edu

Post reply on HN
Print view View original

111–120 of 200 posts

Re: Why are we using black box models in AI when we don’t need to? (2019)

#111

Post by lonelappde »

danShumway wrote: post #91
darkerside wrote: post #76

Earlier quoted context omitted.

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain. AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for usi…

This is tort law, the worst defined laws we have. Liability is far more complex that and usually governed by irrational humans or arbitrary bureaucratic rules contrived by powerful people protecting their interests.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#112

Post by lonelappde »

perl4ever wrote: post #81
MaxBarraclough wrote: post #77

Earlier quoted context omitted.

It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions. That's not the same question as whether there were feelings involved, or whether the process was conscious.

"The point is that the explanation given does not correspond to the process through which the decision was truly reached." Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, o…

Parent wasn't saying that the weighting process was rational. The weights are also feelings.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#113

Post by lonelappde »

bitL wrote: post #99

I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of th…

In your view, it's better to answer the wrong questions optimally than the right questions suboptimally? That's not the kind of AI want putting people in jail or denying them housing or healthcare or shooting missiles at them.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#114

Post by Balgair »

sojournerc wrote: post #85
devit wrote: post #83

Earlier quoted context omitted.

Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.

But that explanation is likely full of cognitive bias, and all sorts of subjective experience. Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.

Object recognition is firmly in the realm of Neuroscience. We've had a good idea what the eye-V1 path is doing in mammals for a few decades now. To the point where we are implanting chips into the brain to (crudely) re-establish vision in blind patients. Though the research is very much on-going, it is being better understood with each passing year.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#115

Post by danShumway »

lonelappde wrote: post #111
danShumway wrote: post #91

Earlier quoted context omitted.

I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain. AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for usi…

This is tort law, the worst defined laws we have. Liability is far more complex that and usually governed by irrational humans or arbitrary bureaucratic rules contrived by powerful people protecting their interests.

But how do black boxes change that? It's still the same people operating under the same laws.

A few commenters here posit that without a black box, designers might be held accountable. If there's a system where software manufacturers would be held accountable for bugs, is any judge going to say, "well, the software is bugged, but they don't know how to fix the bugs, so they're off the hook"?

If there's proof that Twizzlers are poisonous and kill people, the Twizzlers manufacturer can't say, "but we don't know why, we just threw a bunch of chemicals in a vat at random without writing down the labels. So therefore, it's not our fault."

I don't know, can they? I'm not a lawyer, maybe everything in our legal system is way more horrifyingly broken than I assume.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#116

Post by JauntyHatAngle »

fhennig wrote: post #70
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> because despite being less accurate, they can have designers who can be held legally accountable and liable Are you implying you cannot be held accountable for black box models?

An anecdote, but this is a criticism being aimed at YouTube currently for it's demonitisation scheme.

The argument being made is that even though the demonitisation of certain classifications of content is immoral (e.g. demonitising any and all LGBT content, even when family friendly), it is tacitly being allowed to happen due to YouTube's stance of "the algorithm" making the decision, not them.

Which may lead then to have a position of "supporting" LGBT as a company while simultaneously being able to demonitise said content, which could hurt their business in actively hostile to LGBT countries.

This isn't proven, but it's an example of how it could be used to absolve yourself of blame and point it at an inscrutable system.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#117

Post by Balgair »

crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well.

Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though).

I know this may not fit pattern recognition in a super-philosophical sense, but damned if it's not a ways down that road. We're literally shocking a person's brain so that they 'see' again (!!!). How V1 feeds into the rest of the brain is VERY active research and has had a lot of successes, but we're a long ways from a 'black box' these days.

Honestly, the auditory pathways are a lot better understood, as they seem to be 'older' and specific tones are represented physically in the mammalian brain (tonotopy). As such we know a lot better (relative to sight, smell, taste, etc) how tones are encoded and how they elicit specific recognitions and responses (this work is NOT straightforward).

In Neuroscience/Biochem, we are getting a LOT closer, but yes, we are not there yet by any means. The 'black box' idea of the mammalian brain is not going to last another 500 years, likely. We're marching along with good time as a species.

Re: Why are we using black box models in AI when we don’t need to? (2019)

#118

Post by progval »

bitL wrote: post #99

I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of th…

> Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage

While this is probably true (and IMO, possibly the right choice), this:

> faking results of their interpretable models

is taking a huge leap.

Why are they more likely to fake their results than companies selling a black box model?

Re: Why are we using black box models in AI when we don’t need to? (2019)

#119

Post by daoxid »

lonelappde wrote: post #110
daoxid wrote: post #107

Earlier quoted context omitted.

I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.

Introspectable Higher level reasoning is irrelevant. Any introspectable higher level reasoning a person can do has already been automated for efficiency.

Wait, what? Where has this been automated?

Re: Why are we using black box models in AI when we don’t need to? (2019)

#120

Post by username90 »

Balgair wrote: post #117
crazygringo wrote: post #49

I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…

> I think it's important to note that human pattern recognition is basically black-box as well. Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though). I know this may not fit pattern…

Knowing how to input data to the visual cortex doesn't mean that we know how it works. It is the same with neural networks, we know how to pass data to them and how they do the math, but we can't trace back its reasoning to know why it comes to the conclusions it does.
Post reply on HN