EU regulations on algorithmic decision-making and a “right to explanation”
1–10 of 128 posts
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#2Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm...
In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#3So what effect will this have on risk assessment as used by insurance companies?
"Insurance companies use a methodology called risk assessment to calculate premium rates for policyholders. Using software that computes a predetermined algorithm, insurance underwriters gauge the risk that you may file a claim against your policy. These algorithms are based on key indicators about you and then measured against a data set to weigh risk. Insurance underwriters carefully balance the insurance company’s profitability with your potential need to use the policy."
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#4Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
I suppose we can always weasel around with the definition of "why".
Why was my claim refused?
Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud.
Why do those factors indicate fraud?
Because our training data indicates that they do.
Why does your training data indicate that?
...
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#5The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site:
The creditor must tell you the specific
reason for the rejection or that you are
entitled to learn the reason if you ask
within 60 days. An acceptable reason might
be: “your income was too low” or “you haven’t
been employed long enough.” An unacceptable
reason might be “you didn’t meet our minimum
standards.” That information isn’t specific
enough.
(The ECOA is at 15 U.S. Code § 1691, for those who want to go find the gory details)
In one of the lectures from Caltech's "Learning from Data" MOOC the professor mentioned that problem came up at a major lender that he had consulted for. He did not say how they solved the problem.
I wondered if you could generate a satisfactory rejection reason, at least as far as the law is concerned, by taking rejected applications and running them through the system again but with some of the inputs tweaked until they get an acceptance. Then report that the rejection was due to the un-tweaked parameter value.
For instance, if you picked income to tweak, you'd raise the income by, say, $5000 and try again. If that is rejected, raise it another $5000. If it then passes, you can tell the applicant they were rejected because their income was too low, and that an additional $10000 income would be enough to qualify them for a loan.
You'd have to put a bit of sophistication into this to make the explanations reasonable. For instance, if a rejected application would be approved with a very large tweak to income or with a small tweak to employment length it would probably be better to give employment length as the reason for rejection.
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#6I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science.
Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations.
Removing sensitive variables (gender, race, etc) doesn't always help, and specially if the models have high impact on people's lives (applying to loans, university, scholarship, insurance), we cannot afford to blindly use existing annotations in black box models.
All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics.
Edit: here is code for the curious https://github.com/sergioisidoro/aequalis
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#7Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...
Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts.
A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#8The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reaso…
A potential problem with this is that if you "tell the applicant they were rejected because their income was too low" then if this is disputed then they will likely be able to point to many other applicants with even lower income that were accepted, because their total combination of factors was better.
A simple explanation "factor X is too low" implies that there exists a particular cutoff that is required and sufficient; but an explanation that accurately describes why you were rejected would likely be too complex to be understandable.
But yes, probably the intended result could/should be simply identifying the top 1-2 factors that dragged your score down compared to the average accepted candidate (which can be rather easily done automatically even for the most black-box ML methods) and naming them appropriately.
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#9Earlier quoted context omitted.
I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...
Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' f…
Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.
Re: EU regulations on algorithmic decision-making and a “right to explanation”
#10The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?