EU regulations on algorithmic decision-making and a “right to explanation” • Hacker News Board

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#21

Post by throwawaysocks » Thu, Jul 07, 2016, 1:24 PM UTC

crististm wrote: post #12

ddebernardy wrote: post #2

Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?

You are certainly right to question the reasoning behind AI decisions. I'm not sure that neural network based AI can do that though. For reference see an older thread on this very topic: https://news.ycombinator.com/item?id=10388795

> I'm not sure that neural network based AI can do that though.

Generating human-readable explanations from a machine learning algorithm or NN is probably an impossible problem to solve in full generality.

However, for many particular systems, doing so should be feasible.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#22

Post by dozzie » Thu, Jul 07, 2016, 1:32 PM UTC

tzs wrote: post #5

The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reaso…

There was this idea (I think I stumbled upon it on Bruce Schneier's blog) where the algorithm used to judge your credit score was actually public, and there was a way to dispute its outcome if you felt it was unfair in your case.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#23

Post by throwawaysocks » Thu, Jul 07, 2016, 1:37 PM UTC

AnthonyMouse wrote: post #19

sergioisidoro wrote: post #6

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and sp…

> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time trav…

> maximizing prediction accuracy is inherently unbiased

This assumes that you're actually maximizing prediction accuracy, rather than taking the easiest route toward sufficiently high predictive power.

Not-so-hypothetical: you can invest $N and create a profitable model that (unfairly and inaccurately) discriminates directly based upon race, or you can invest $N*M and create a profitable model that does not discriminate on race (regardless of whether its results are racially equitable). Given the choice, a lot of people will choose the $N approach.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#24

Post by phkahler » Thu, Jul 07, 2016, 1:39 PM UTC

zoner wrote: post #17

Finally I get to know why my mortgage was declined :)

>> Finally I get to know why my mortgage was declined :)

My first attempt at one was declined due to "problems with my credit report" or some such. I happened to be working at a place where they did credit checks on customers, so I asked someone to pull my credit report. They had all my stuff, but had mixed in a bunch of information from someone else with the same name - different age, there were loans on there from when I was 3. I called the reporting company to complain and they kept getting caught up in "how did you get your report?" as if I had done something wrong in just having access to their incorrect information. Perhaps having someone pull the info instead of the "consumer" getting it through proper channels was against some rule - this was 1995. I think the same is often true today, companies want to collect data on you but don't really want you to know what they've got or how they use it.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#25

Post by dharma1 » Thu, Jul 07, 2016, 1:41 PM UTC

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech.

There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of the data, or offer manual review of the application if rejected (which is something they do) - but it would be very hard to show exact causal reasoning patterns in the automatic scoring.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#26

Post by tombone12 » Thu, Jul 07, 2016, 1:45 PM UTC

AnthonyMouse wrote: post #19

sergioisidoro wrote: post #6

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and sp…

> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time trav…

You entirely miss the point! The point is that in supervised learning for example, if you optimize prediction accuracy with respect to your human generated examples, you will get a model that exactly reproduces the racist judgment of the human that generated your training set.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#27

Post by dec0dedab0de » Thu, Jul 07, 2016, 1:46 PM UTC

dharma1 wrote: post #25

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of th…

why not just have an optional pdf download with the 20000 vectors, and how much they were weighted.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#28

Post by fixermark » Thu, Jul 07, 2016, 1:48 PM UTC

PeterisP wrote: post #14

fixermark wrote: post #9

Earlier quoted context omitted.

Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.

If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal. However, if you do some reason analysi…

This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it.

"Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#29

Post by GFK_of_xmaspast » Thu, Jul 07, 2016, 1:49 PM UTC

dharma1 wrote: post #25

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of th…

Sounds like a limitation of their business model.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#30

Post by kaijush » Thu, Jul 07, 2016, 1:49 PM UTC

dharma1 wrote: post #25

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of th…

>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable?

Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.