Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EU regulations on algorithmic decision-making and a “right to explanation”

arxiv.org

Post reply on HN
Print view View original

61–70 of 128 posts

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#61

Post by Symmetry »

yoo1I wrote: post #55
hx87 wrote: post #48

Earlier quoted context omitted.

Then the problem is in the racist human(s), not the algorithm.

But the algorithm enshrines and possibly amplifies it. Or as the old saying goes: > To err is human, to really foul things up requires a computer.

Humans have much more potential for racism in making predictions about the future than in judging things that have already happened. So while I agree that racism through biased input data is a problem, I think that even with that problem machines should be substantially less racist in their judgement than the humans they're replacing even if they're not perfect.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#62

Post by jacquesm »

sergioisidoro wrote: post #59
InTheArena wrote: post #39

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This i…

I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)

Interesting aside (but still On Topic): This works for IP addresses as well as zip codes.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#63

Post by Symmetry »

I wonder machine learning could just handle this the way that humans do. Human decisions processes are of course extremely opaque even if we're good at coming up with post-hoc justifications for our reasoning as needed.

"You're income is too low" "But his income is also low and he got a loan!" "Well he has a better credit score." etc

It would obviously be impossible to give a complete and accurate explanation of a machine learning decision process the same way it would be impossible for a human learning decision process, but I wonder if it would actually be difficult to come up with particularly salient distinguishing features in general or between certain cases the same way that people do?

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#64

Post by Symmetry »

matt4077 wrote: post #11
mknocker wrote: post #10

The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?

Nope, because advertisement doesn't "significantly affect" you. At least that's how it's meant and how it will be understood. This is about credit, employment, insurance, medical care etc.

Medical care?! Oh God. Well, I suppose the AMA will keep the US from embracing machine learning so it's not like this is actually a chance for the US to reverse its lag in mortality statistics.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#65

Post by asherkin »

zoner wrote: post #54
dublinben wrote: post #31

Earlier quoted context omitted.

This is exactly why you are legally entitled to receive a copy of your credit report every year. You can challenge anything on it, and have incorrect information removed.

Not in the EU (or at least not in The UK where I currently live).

Yes you are, it'll cost you £6 in processing fees to go to all 3 agencies.

http://www.experian.co.uk/consumer/statutory-report.html

https://www.equifax.co.uk/Products/credit/statutory-report.h...

http://www.callcredit.co.uk/consumer-solutions/your-credit-r...

You can alternatively use the free online services:

Noddle use data from Callcredit, and Clearscore use data from Equifax - I don't think there is a free service exposing Experian data.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#66

Post by tomaskafka »

InTheArena wrote: post #39

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This i…

Yep. Or other -isms:

> What if [your hiring algorithm] is weeding out women most likely to be pregnant in the next year?

https://theoverspill.wordpress.com/2016/07/04/start-up-linke...

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#67

Post by yummyfajitas »

sergioisidoro wrote: post #59
InTheArena wrote: post #39

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This i…

I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)

But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores.

In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive.

I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#68

Post by tomaskafka »

dharma1 wrote: post #47
kaijush wrote: post #30

Earlier quoted context omitted.

>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.

They validate the accuracy of the model with real users, starting with short term/low value loans, to mitigate risk. The explanation for them is - it works.

"How would you find out that it doesn't work? For a specific group of people for example?"

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#69

Post by robrenaud »

InTheArena wrote: post #39

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This i…

I think there are two problems with requiring explainable models:

Explainable models will be more easily gamed, and they are likely to be less accurate.

The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A and B. They go and fix reason A and B, but in the mean time, so have many other credit seekers, diminishing their predictive value. In that time, the modelers create a new, different explainable model, that no longer uses A and B, but somewhat related predictors C and D, which haven't yet been used, and so haven't yet been gamed, which the original seeker does not meet?

Explainable models, being a subset of all models, are likely to not contain the most accurate models. I don't anything about the domain of credit scoring (maybe state of the art models are actually small and understandable?), but in speech recognition, for example, models are constantly growing in complexity, their individual decisions are way beyond explainable to anyone in reasonable amount of time, and they are only getting more powerful as they get larger and more complex. In speech, models are already many gigs. In credit scoring, less accurate models mean higher rates, so there is an aggregate loss.

[1] https://en.wikipedia.org/wiki/Campbell%27s_law

Re: EU regulations on algorithmic decision-making and a “right to explanation”

#70

Post by wodenokoto »

AnthonyMouse wrote: post #52
tombone12 wrote: post #26

Earlier quoted context omitted.

You entirely miss the point! The point is that in supervised learning for example, if you optimize prediction accuracy with respect to your human generated examples, you will get a model that exactly reproduces the racist judgment of the human that generated your training set.

> The point is that in supervised learning for example, if you optimize prediction accuracy with respect to your human generated examples, you will get a model that exactly reproduces the racist judgment of the human that generated your training set. In which case you aren't optimizing prediction accuracy. Prediction accuracy is measured by whether the predictions are true. If you have bias in the predictions which d…

You need to think about how we measure truth and even what truth is.

In machine learning we tend to assume the annotations and labels are "true" and build a system towards that version of the "truth".

> Prediction accuracy is measured by whether the predictions are true.

The more I think about this sentence, the less sense it makes. Prediction accuracy can only be measured against records of something, and that record will be a distortion and simplification of reality.

Post reply on HN