Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history and future of credit scores

economist.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: A brief history and future of credit scores

#31

Post by mindslight »

dahfizz wrote: post #29
emadelwany wrote: post #9

Earlier quoted context omitted.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero.

The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all, regardless of it seeming quite lucrative to hire a compliance department to analyze just how far you can get away with stretching it.

Re: A brief history and future of credit scores

#32

Post by skybrian »

swsieber wrote: post #17

There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?

Maybe the idea is that banks should have an incentive to find some better data that isn't a proxy for race? That is, getting closer to the ideal of judging people as individuals instead of as a members of prohibited groups.

Re: A brief history and future of credit scores

#33

Post by jcranmer »

dahfizz wrote: post #29
emadelwany wrote: post #9

Earlier quoted context omitted.

Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

> How removed from race does information have to be in order to be considered by a financial firm?

In general, the standard is "disparate impact"--if you accepted 80% of all white applicants but only 20% of black applicants, then you're probably liable for racial discrimination even if you were completely race-blind.

Re: A brief history and future of credit scores

#34

Post by adaptiveValleys »

skybrian wrote: post #32
swsieber wrote: post #17

There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?

Maybe the idea is that banks should have an incentive to find some better data that isn't a proxy for race? That is, getting closer to the ideal of judging people as individuals instead of as a members of prohibited groups.

I think that's the intent, but it seems to only have incentivized discovering another proxy for race. For example, I recently watched an infosec talk 'hacking your credit score.'[0] Where the presenter mentioned that Fair Isaac (a reporting agency mentioned in the article) has a parameter in their algorithm called 'HMA' (High Minority Area), that he found in an internal presentation. I think a solution would be any parameter that remotely gives a correlate for race should be entirely eliminated from scoring systems.

Another startling insight from that talk was how normal services (i.e. home utilities, insurance, etc) will do credit inquiries and set a customer's rate based on their score. Which results in people with low credit scores pay more for services that are traditionally unrelated to borrowing. I'm worried that it could lead to a positive feedback loop that heavily affects those with poor credit scores in the long term. For this, it seems like a limitation on the kind of business relationships allowed to perform credit inquiries should be implemented.

0. https://www.youtube.com/watch?v=5gFDnQGr6WU

Re: A brief history and future of credit scores

#35

Post by throwawaywego »

swsieber wrote: post #17

There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?

There is multiple elements to your question.

Two ways where these systems may give out more loans than is strictly profitable, and they are both investments:

- Fairness. If you have a variable race and a zip code, you could account for discrimination via redundant encodings, while still using the feature for the optimal trade-off between a fairness criteria and model performance.

- Exploration. Concept drift (the correlational and causal meaning of variables shifts over time) can introduce wrong predictions. If all you have is few samples from a zip code, the model will always be uncertain. You can counter this by exploration and active learning: gather samples, not because this makes the model max-profit, but gather samples, to better learn how to predict these samples in the future.

But yes, giving out too many loans to minorities, may very well lead to further crisis and defaults, and tainting the credit scores of people will low access to finances even further. A bit like how well-meaning people donate money and food to Africa during Christmas time, then a few months later when donations subside, there are increases in famines. There is such a thing as "being too good".

Re: A brief history and future of credit scores

#36

Post by throwawaywego »

diminoten wrote: post #3

The key here being that non-financial data isn't actually useful in predicting ability to repay loans. I'm sure it'd be used somehow by modern financial institutions if it were predictive.

> The key here being that non-financial data isn't actually useful in predicting ability to repay loans.

Sure it is. Even religion correlates with loan risk. (Could be a proxy for social status and people from your tribe helping you out when you can't pay back). I could probably get a predictive model better than random guessing by mining your HackerNews comments or Facebook likes.

> I'm sure it'd be used somehow by modern financial institutions if it were predictive.

It is used. All data that is even remotely informative is used. To the fullest extend made possible by jurisdiction/ anti-discrimination laws.

Re: A brief history and future of credit scores

#37

Post by throwawaywego »

dd36 wrote: post #6
dvdbloc wrote: post #5

Earlier quoted context omitted.

I find this interesting because when I applied for a mortgage at Wells Fargo they asked me for my race in the application process.

I get confused by those they claim some of these ML or NN algos are black boxes and so could be breaking the law. If you’re not inputting illegal info (like race, sex, national origin, religion, name, etc. or corollaries), then it’s not making its risk assessment on that basis. All you have to do is look at the inputs. It isn’t unknowable whether or not it’s breaking the law.

Other comments have pointed out the risk of redundant encodings /proxy variables.

I like to point out that sources of unfairness are possible, even with 100% decorrelated non-protective variables. For instance, the data collection and labeling of the data may be biased (You label re-admission to jail with current criminals in jail, and just overfitted to the war on drugs).

Or you make a biased decision based on the output of a fair model. Things like not taking into account sample size / uncertainty.

Also, breaking the law is at odds with unethical behavior. Unethical behavior is not necessarily breaking the law, but it is still nasty. For instance, from your Facebook likes (not a protected variable) I could deduce all of race, sex, origin, religion. Not against the law, but still discrimination.

Re: A brief history and future of credit scores

#38

Post by lifeisstillgood »

btown wrote: post #26
swsieber wrote: post #17

There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?

This brings up a bit of a "trolley problem" of an ethical dilemma. Say you're asked to create a machine learning system, but you know that the data quality is so poor that you're very likely to overfit in a way that will deny economic opportunities to underserved communities that currently have them. But if you don't create that system, you're denying economic opportunities to other underserved communities that curre…

>>> no Hippocratic oath for our profession

Sadly there is no profession for our profession. I often think the model for any software profession (if we can create that - something I doubt) is railway engineer - where the professional signs off on the safety / completeness of work done on the railway - and that no train can travel without it. It leads to plenty of uncompetitive practises - but also to ... y'know ... people not dying in crashes.

How we start that is hard (probably something to do with safety critical software systems) because we aren't too sure what is the right way to build software.

And then we have the fun problem of the members of the profession trying to decide the answer to your trolley problem. Sorry scratch that. The various legislatures proscribing the answer and the profession trying to implement the conflicting results !

Re: A brief history and future of credit scores

#39

Post by lifeisstillgood »

I often think the most effective means to increase loan repayment rates is to provide the debtor with a effective accurate money management tool - sort of like Mint but better. I am supposedly a well educated intelligent software engineer and yet trying to get a single unified view of what I spent is outrageously challenging or requires discipline at a level of dieting.

Apart from fraudsters, people who take out a loan want to pay it back but like dieting human failings cause the problems.

Just a instant check on what you have spent globally will make a huge difference in budget management.

Re: A brief history and future of credit scores

#40

Post by tareqak »

jetrink wrote: post #16
mieseratte wrote: post #12

Earlier quoted context omitted.

Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?

A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon'…

From my reading of that article, I think the recruiting tool was fed resumés and a data point saying whether or not the corresponding candidate was hired or not. As a result, the tool not only developed a bias against women, but was effectively evidence that there was bias against women in the original hire / not hire decisions.
Post reply on HN