EU regulations on algorithmic decision-making and a “right to explanation” • Hacker News Board

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#81

Post by dublinben » Thu, Jul 07, 2016, 4:48 PM UTC

sshumaker wrote: post #56

jerf wrote: post #45

Earlier quoted context omitted.

When mentioning this, I find it helpful to include this page: https://www.ftc.gov/faq/consumer-protection/get-my-free-cred... If you are one of the people just finding out about this today, and you Google "free credit report", you come up with a loooot of bad, scammy links. This is how you get your legally-mandated, really free, annual credit report, starting from an ftc.gov address so you know it's really the right…

Credit Karma is legit and free. Also gives you your credit score and credit monitoring (all free). (Disclaimer: I work there).

To be fair, the credit score (VantageScore 3) you provide is not the credit score (FICO) most people are interested in, and that is used for most credit decisions.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#82

Post by basseq » Thu, Jul 07, 2016, 4:50 PM UTC

PeterisP wrote: post #8

tzs wrote: post #5

The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reaso…

A potential problem with this is that if you "tell the applicant they were rejected because their income was too low" then if this is disputed then they will likely be able to point to many other applicants with even lower income that were accepted, because their total combination of factors was better. A simple explanation "factor X is too low" implies that there exists a particular cutoff that is required and suffi…

> an explanation that accurately describes why you were rejected would likely be too complex to be understandable.

This is the core of the problem. Have you ever tried to explain a statistical segmentation to someone? It requires them to let go of how they think about if-x-then-y criteria and attributes. Very intelligent people who understand statistics, weighting, clustering, etc. struggle with it--think about an "average" person.

You can back into a reason: "Here's the attribute(s) on which you deviate the most from the median of the segment of which you want to be a part." But it's not a linear, single-variable "fix".

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#83

Post by Houshalter » Thu, Jul 07, 2016, 4:51 PM UTC

vkou wrote: post #78

InTheArena wrote: post #74

Earlier quoted context omitted.

True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.

The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.

The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#84

Post by kaijush » Thu, Jul 07, 2016, 4:52 PM UTC

JoeAltmaier wrote: post #43

kaijush wrote: post #30

Earlier quoted context omitted.

>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.

Because the training set was covered with x% accuracy. Ok, how does knowing that help?

OK, so we're assuming supervised learning. In that case, how were the features of the training set chosen? And how was the training set labelled?

Most of the time, outside of semi-supervised learning, those things are not in the data, someone has to (painstakingly) decide them. That can very well be explained I believe.

More generally, what I mean to say is that you can learn any number of models from the same data, so to a great extent you choose what your model learns by manipulating your algorithm's (hyper)parameters, the training set's features, choosing what goes into the training set itself etc etc. That process leaves room for a lot of regulatory oversight.

- in principle. In practice I expect since we're talking about people lending money or selling insurance we're only ever going to get those details out of their cold dead hands, and only with considerable effort at that.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#85

Post by basseq » Thu, Jul 07, 2016, 4:57 PM UTC

dozzie wrote: post #22

tzs wrote: post #5

The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reaso…

There was this idea (I think I stumbled upon it on Bruce Schneier's blog) where the algorithm used to judge your credit score was actually public , and there was a way to dispute its outcome if you felt it was unfair in your case.

Interesting from an academic perspective, but absolutely inaccessible to 99% of the population. If I don't get the outcome I want, I will dispute it (and people already do) because I think it's unfair.

And maybe it is in your case. Are all algorithms designed to be "fair"? Should they be? Can they be, if everyone has an exception or a loophole? Or at some point do "disputes" go back to a human with the same biases that algorithms were intended to solve?

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#86

Post by basseq » Thu, Jul 07, 2016, 4:59 PM UTC

throwawaysocks wrote: post #23

AnthonyMouse wrote: post #19

Earlier quoted context omitted.

> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time trav…

> maximizing prediction accuracy is inherently unbiased This assumes that you're actually maximizing prediction accuracy, rather than taking the easiest route toward sufficiently high predictive power. Not-so-hypothetical: you can invest $N and create a profitable model that (unfairly and inaccurately) discriminates directly based upon race, or you can invest $N*M and create a profitable model that does not discrimin…

> regardless of whether its results are racially equitable

Some would call this discrimination, if not outright racism.

As others on this thread have pointed out, at some point the algorithm follows the underlying data, which may also be discriminatory (zip code, etc.).

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#87

Post by basseq » Thu, Jul 07, 2016, 5:04 PM UTC

tomaskafka wrote: post #66

InTheArena wrote: post #39

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This i…

Yep. Or other -isms: > What if [your hiring algorithm] is weeding out women most likely to be pregnant in the next year? https://theoverspill.wordpress.com/2016/07/04/start-up-linke...

This is the law of unintended consequences. Data points are correlated to other data points. Unless your hiring algorithm is a random number generator, you could find some sort of "bias" you didn't intend.

At least an algorithm is measurable, repeatable, and consistent. Yes, you have to monitor the output of any process for unintended consequences, then you make tweaks as appropriate to eliminate unintended outcomes.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#88

Post by yummyfajitas » Thu, Jul 07, 2016, 5:10 PM UTC

Houshalter wrote: post #83

vkou wrote: post #78

Earlier quoted context omitted.

The stereotype (That predominantly black households in a poor area code are poorer then the predominantly white households in a rich area code) can be proven correct by simple demographics, and the fact that household wealth is correlated with credit score. The problem is that there's a positive feedback loop at play, here.

The bank already knows their assets and income. The question is if an equally poor and educated white is just as likely to repay a loan as an equally poor and educated black. I imagine most of the difference would go away. Unless you really believe black people are inherently less likely to pay back loans, all else equal.

If you can come up with an algorithm that reproduces this conclusion - with accuracy even remotely close to the "racist" ones - banks will cross heaven and earth to pay you $billions.

Unfortunately the only public analysis I'm aware of is from a blog using Zillow data: https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t...

This effect is reproduced in various walks of life, e.g. education.

You'll be doing social good in addition to earning billions of dollars. What's not to like?

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#89

Post by yummyfajitas » Thu, Jul 07, 2016, 5:16 PM UTC

InTheArena wrote: post #72

robrenaud wrote: post #69

Earlier quoted context omitted.

I think there are two problems with requiring explainable models: Explainable models will be more easily gamed, and they are likely to be less accurate. The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A…

A fair point, but as a society, we have decided that racial discrimination is not a valid mechanism for banks to profit by. That does result in everyone paying a bit more in interest as the risk pool is larger, but a acceptable tradeoff. In terms of gaming, verification is just as important as scoring If the data you have going into to the system is rigged, and income is not being properly validated, bad things will…

As a society we have directed banks to make bad loans to blacks and charge non-blacks extra to make up the difference? I'd be surprised if even 10% of people know this decision was made.

Also, what makes it acceptable to engage in this form of surreptitious wealth redistribution on racial lines?

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#90

Post by csn » Thu, Jul 07, 2016, 5:20 PM UTC

matt4077 wrote: post #11

mknocker wrote: post #10

The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?

Nope, because advertisement doesn't "significantly affect" you. At least that's how it's meant and how it will be understood. This is about credit, employment, insurance, medical care etc.

Although just like the things you list, I can think of a plethora of ways with which advertising can literally, indirectly change my life by convincing me to purchase a product or service.