Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists Increasingly Can’t Explain How AI Works

vice.com

Post reply on HN
Print view View original

11–20 of 231 posts

Re: Scientists Increasingly Can’t Explain How AI Works

#11

Post by dmichulke »

croes wrote: post #4

To me the problem of AI is not how it works but how reliable it works. Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?

What you're saying is (kinda) my response when people ask me about self-driving cars:

How does the car deal with object X on the road? For X=piano, a stack of solar panels, a tank, an airplane, a pile of stones ...

No one knows and since the size of the set of X is infinite, no one can appropriately train for it either.

That's why we can't have self-driving cars without a general understanding of what objects are and how they move.

Re: Scientists Increasingly Can’t Explain How AI Works

#12

Post by bobosha »

I think explainability is overrated (to use a Trumpian expression) & this fixation with coming up with explanations for inferences is a red herring. We cannot explain our own thoughts and actions and tend to ascribe logic & reason to many of our own actions, but it's almost always system-1 driven, for the most part. Correcting for edge-cases and unknown-unknowns is where we should focus our efforts methinks.

Re: Scientists Increasingly Can’t Explain How AI Works

#13

Post by joxel »

It's quite easy to explain. You take a bunch of tensors and multiply them by a bunch of tensors.

Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.

Re: Scientists Increasingly Can’t Explain How AI Works

#14

Post by squaredot »

FartyMcFarter wrote: post #7

> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers. In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-styl…

I think these kinds of articles are geared toward people who do not know (even superficially) about the models. Perhaps the sentiment they want to elicit is "If the person who created these models cannot explain how they work, then he is not so different from me."

Re: Scientists Increasingly Can’t Explain How AI Works

#15

Post by pixelfarmer »

croes wrote: post #4

To me the problem of AI is not how it works but how reliable it works. Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?

I read neural network (NN) books in the 90ies, and they covered for example the problem with corner cases already. One thing I remember is a (comparatively simple) NN which created the past tense of an english verb, and there are exceptions like "go" and "went". It preserved that for a while, but further training brought it back to "goed", because NNs do generalize and you need tons of additional "storage" in a NN to keep such corner cases around. That also means if you need (near) 100% reliability you are in the wrong place when it comes to NN aka "AI".

Re: Scientists Increasingly Can’t Explain How AI Works

#16

Post by joxel »

croes wrote: post #4

To me the problem of AI is not how it works but how reliable it works. Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?

I mean no responsible developer would take the output of a ML model and then act on that without any tests on whether the output makes sense or not. That's the way to handle unexplainable AI.

There's also opportunity to look at uncertainty estimation, i.e. look at the epistemic uncertainty of the model and use it as a proxy for potential error. That seems to be a main thrust for AI right now. I'm currently writing a paper looking at this for dynamical systems.

Re: Scientists Increasingly Can’t Explain How AI Works

#17

Post by mdp2021 »

FartyMcFarter wrote: post #7

> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers. In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-styl…

> anything new

It is nothing new. It is called (at least) "the problem of transparency".

The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info).

It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals.

And of course, "understanding" the engine is required to advance it.

It is a key problem.

Re: Scientists Increasingly Can’t Explain How AI Works

#18

Post by photochemsyn »

The history of scientific development is one of finding patterns in data (think Kepler's studies of Tycho Brahe's accurate astronomical observations) - yes, AI excels at that - but then expressing those patterns in terms of simple mathematical equations: orbits are elliptical to a first approximation (not counting other graviational bodies than the sun and a planet), and the sun->planet vector sweeps out equal areas in equal times, etc.

https://www.physicsclassroom.com/class/circles/Lesson-4/Kepl...

AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?

Re: Scientists Increasingly Can’t Explain How AI Works

#20

Post by fallingknife »

Previous technology has observed things that people can't by essentially increasing the power of one of our senses, e.g. a microscope increases visual resolution, or a radio telescope increases visual range. This is very easy for the brain to understand. But if you can build a machine to improve on the human brain itself in ability to recognize patterns, is it even possible to get the brain to understand it?
Post reply on HN