There's a pretty cool paper that visualises the different layers in a ConvNet trained on ImageNet: https://arxiv.org/pdf/1311.2901.pdf - page 4 has the layer imagery.
For AI to thrive, it must explain itself
31–40 of 72 posts
Re: For AI to thrive, it must explain itself
#32I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it. In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interp…
I cannot explain how I form the sentences I speak. I still find speaking useful.
Re: For AI to thrive, it must explain itself
#33Earlier quoted context omitted.
I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).
It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve
Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent.
I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them.
Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.
Re: For AI to thrive, it must explain itself
#34Some people are missing the point.
> "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law."
It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates.
> "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful or discriminatory results." ... " If the randomisation of sex produces no change in the number of women offered jobs by the AI, but randomising weightlifting ability increases it (because some women now appear to have “male”abilities to lift weights), then it is clear that weightlifting ability itself, not an applicant’s sex, is affecting the hiring process."
This is the most interesting part of the article. As it shows that it's possible to test the system for unlawful decisions without actually understanding how it thinks.
Re: For AI to thrive, it must explain itself
#35I’d like to simply know “how good is AI right now” at any given time. This information seems largely hidden. For example, can any AI expert tell me: what is the exact date when Target corporation crossed the threshold of being able to face match 20% of its customers across different stores with 20% accuracy? How about 50% (for both of the different numbers, to keep it simple). How about 80%? 90%? That’s just one exam…
These are published in papers, but for real-world products, they're competitive advantage. Why should we share them with anyone other than our customers?
Data for very specific narrow tasks like face recognition is published. How the capabilities are combined in a special sauce, not so much.
So yeah, going off what you said, all these people saying “don’t worry, nothing to worry about with AI” haven’t the slightest clue what AI is up to outside of whatever bubble they are in.
Re: For AI to thrive, it must explain itself
#36Earlier quoted context omitted.
There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place. Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your acces…
If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".
Very much not.
Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence.
The software has no more agency than a cruise control.
Re: For AI to thrive, it must explain itself
#37Earlier quoted context omitted.
If those elements of the graph are "spurious", why not remove them from the very beginning? Or, should I say, not have them there?
Because we dont know which ones are spurious and the parent is pretty optimistic about the idea of being able to analyize which ones are/are not.
We only know which ones are spurious if we run the trimming algorithms. I wrote a paper about 10 years ago showing how to use generic algorithms on the weight matrix to remove spurious interactions.
Re: For AI to thrive, it must explain itself
#38Some people are missing the point. > "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law." It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates. > "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful…
It sounds like a useful test if you get a positive result for discrimination. If the test comes back negative, things are more complicated.
This isn't as simple as it seems because a supposed job requirement, while superficially neutral, may be unnecessary. For example, police departments used to have height requirements. But height requirements discriminate against women. Do you need to be tall to be a police officer? Maybe it's really about projecting authority? Okay, how do you measure that?
Making job requirements neutral is a rather fuzzy area. What is the job really about? Who decides?
Any variable that's not independent of gender can be used to (imperfectly) detect gender. It may not be easy to figure out if machine learning is using it for something real or just as a substitute gender-detector. Randomizing gender while leaving everything else the same isn't going to detect this.
Re: For AI to thrive, it must explain itself
#39Some people are missing the point. > "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law." It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates. > "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful…
AI models simply exhibit the behaviour of the data they are trained on. If you make a model that tries to copy an individual persons behaviours, is it the AI model that is at fault or the person it's modelling? Where should the liability lay?
Re: For AI to thrive, it must explain itself
#40Earlier quoted context omitted.
Because we dont know which ones are spurious and the parent is pretty optimistic about the idea of being able to analyize which ones are/are not.
We only know which ones are spurious if we run the trimming algorithms. I wrote a paper about 10 years ago showing how to use generic algorithms on the weight matrix to remove spurious interactions. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/
[deleted]