Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EU regulations on algorithmic decision-making and a “right to explanation”

arxiv.org

Post reply on HN
Print view View original

11–20 of 128 posts

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#11

Post by matt4077 »

mknocker wrote: post #10

The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?

Nope, because advertisement doesn't "significantly affect" you. At least that's how it's meant and how it will be understood. This is about credit, employment, insurance, medical care etc.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#12

Post by crististm »

ddebernardy wrote: post #2

Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?

You are certainly right to question the reasoning behind AI decisions. I'm not sure that neural network based AI can do that though.

For reference see an older thread on this very topic: https://news.ycombinator.com/item?id=10388795

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#13

Post by crististm »

PeterisP wrote: post #7
chriswarbo wrote: post #4

Earlier quoted context omitted.

I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...

Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' f…

Maybe Eric Shmidt is a lower risk customer because he's opinion on "don't be evil" is that is a bad idea? :)

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#14

Post by PeterisP »

fixermark wrote: post #9
PeterisP wrote: post #7

Earlier quoted context omitted.

Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' f…

Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.

If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal.

However, if you do some reason analysis, then this is useful and gives a lot of interesting information.

Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc.

Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available.

Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability.

I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#15

Post by matt4077 »

PeterisP wrote: post #7
chriswarbo wrote: post #4

Earlier quoted context omitted.

I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...

Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' f…

If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them.

I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#16

Post by PeterisP »

matt4077 wrote: post #15
PeterisP wrote: post #7

Earlier quoted context omitted.

Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' f…

If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering abou…

I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#18

Post by denzil_correa »

mknocker wrote: post #10

The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?

There is some research which shows discrimination in online advertisements.

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2208240

http://queue.acm.org/detail.cfm?id=2460278

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#19

Post by AnthonyMouse »

sergioisidoro wrote: post #6

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and sp…

> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations.

Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time travel to tell you with 100.0% accuracy who would pay back their loans, there would be a racial disparity in the result, but the cause of it is not the fault of the algorithm.

And you can't fix it there because that's not where the problem is.

Suppose you have a group of 800 white middle managers, 100 white sales clerks and 100 black sales clerks. Clearly the algorithm is going to have a racial disparity in outcome if the middle managers are at 500% of the cutoff and the sales clerks are right on the line, because it will accept all of the middle managers and half of the sales clerks which means it will accept >94% of the white applicants and 50% of the black applicants.

But the source of the disparity is that black people are underrepresented as middle managers and overrepresented as sales clerks. The algorithm is just telling you that. It can't change it.

And inserting bias into the algorithm to "balance" the outcome doesn't actually do that, all you're doing is creating bias against white sales clerks who had previously been on the same footing as black sales clerks. The white middle managers will be unaffected because they're sufficiently far above the cutoff that the change doesn't affect them, even though that they're the source of the imbalance.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#20

Post by throwawaysocks »

PeterisP wrote: post #16
matt4077 wrote: post #15

Earlier quoted context omitted.

If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering abou…

I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.

Definitely.
Post reply on HN