http://archive.is/NtfoT
Thanks, economist.com is terrible
A brief history and future of credit scores
21–30 of 72 posts
Re: A brief history and future of credit scores
#22There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?
Yes, it would lead to more defaults, but it would also lead to fewer false negatives (people denied loans who would have paid back).
Re: A brief history and future of credit scores
#23Earlier quoted context omitted.
Using zip codes to implement discriminatory policies isn’t ML, but has almost certainly occurred. It’s referred to as redlining, and has been around for a while. The New Deal had some pretty indefensible redlining conditions. A lot of people will say the practice continues to this day, but the modern examples are a lot more open to interpretation than some of the historical ones are.
Was the New Deal redlining out-and-out discrimination or a matter of accidentally creating conditions, i.e. based on bad intelligence, that resulted in discrimination. I always understood it to be the former.
The level of plausible deniability regarding The New Deal is mostly a matter of opinion, but personally I think it was pretty transparent in how it intentionally discriminated.
It’s the modern examples that I think are much more questionable. For example, there are more liquor stores in black neighbourhoods. Is this because:
a) A conspiracy to use alcohol to suppress the black population?
b) Those neighbourhoods have a greater demand for liquor stores?
Re: A brief history and future of credit scores
#24[paywalled]
Re: A brief history and future of credit scores
#25Earlier quoted context omitted.
A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon'…
I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.
This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-cons, whatever.) However, some of the patterns that they detect are our biases against classes of people. If the solution is to sanitize the data to such a degree that the ML system can no longer infer that someone is a woman or that someone is over 40, etc., then the training data is probably also useless for detecting the non-obvious patterns that we want the system to discover.
Re: A brief history and future of credit scores
#26There's commentary abiut incidentally discriminating based on race (e.g. zip code aa an input can act as a proxy for race). Would giving out more loans than is rational by excluding stuff like zip codes be a good thing? Wouldn't that lead to more defualts among those groups of people zip codes can discriminate against?
This brings up a bit of a "trolley problem" of an ethical dilemma. Say you're asked to create a machine learning system, but you know that the data quality is so poor that you're very likely to overfit in a way that will deny economic opportunities to underserved communities that currently have them. But if you don't create that system, you're denying economic opportunities to other underserved communities that currently do NOT have them. Do you take the job? Moreover, if you'll do less harm than someone else who might be hired, does that make a difference?
There's no Hippocratic oath for our profession, and in many ways that's important, because we create systems whose impact may very well outlive us and out-scale anything that a single medical professional could do. But that also doesn't mean we should operate in a utilitarian environment without constraints.
Re: A brief history and future of credit scores
#27Earlier quoted context omitted.
I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.
This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-co…
That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples.
Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected information in a roundabout way.
Re: A brief history and future of credit scores
#28The key here being that non-financial data isn't actually useful in predicting ability to repay loans. I'm sure it'd be used somehow by modern financial institutions if it were predictive.
I find this interesting because when I applied for a mortgage at Wells Fargo they asked me for my race in the application process.
Yes I’ve used cfpb hmda data and race is a column in reported lending.
Re: A brief history and future of credit scores
#29Earlier quoted context omitted.
I get confused by those they claim some of these ML or NN algos are black boxes and so could be breaking the law. If you’re not inputting illegal info (like race, sex, national origin, religion, name, etc. or corollaries), then it’s not making its risk assessment on that basis. All you have to do is look at the inputs. It isn’t unknowable whether or not it’s breaking the law.
Let me try to clear your confusion: an input my seem innocent (e.g. zip code), but a zip code is likely to correlate to ethnicity and race in some regions. So even if the inputs seem legal, an ML model that’s sophisticated enough can derive illegal results that discriminate against certain populations.
This gets at the heart of one of the issues with these discrimination laws.
What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact?
How removed from race does information have to be in order to be considered by a financial firm?
Re: A brief history and future of credit scores
#30Earlier quoted context omitted.
Are there any real-world examples of an ML system causing protected-class discrimination based on non-protected criteria?
A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon'…
Can you give another example?