Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A brief history and future of credit scores

economist.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: A brief history and future of credit scores

#51

Post by Applejinx »

adaptiveValleys wrote: post #34
skybrian wrote: post #32

Earlier quoted context omitted.

Maybe the idea is that banks should have an incentive to find some better data that isn't a proxy for race? That is, getting closer to the ideal of judging people as individuals instead of as a members of prohibited groups.

I think that's the intent, but it seems to only have incentivized discovering another proxy for race. For example, I recently watched an infosec talk 'hacking your credit score.'[0] Where the presenter mentioned that Fair Isaac (a reporting agency mentioned in the article) has a parameter in their algorithm called 'HMA' (High Minority Area), that he found in an internal presentation. I think a solution would be any p…

I don't think 'could lead to' is correct, because that implies this can only happen in a structured, intentional way.

It's an epiphenomenon of the interaction between this information and the normal workings of society on so many levels, and that feedback loop has been in place for a LONG time and is only getting stronger, now with real intentionality.

Re: A brief history and future of credit scores

#52

Post by michaelt »

mieseratte wrote: post #27
jetrink wrote: post #25

Earlier quoted context omitted.

This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-co…

That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples. Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected informatio…

Age is a protected class, and it's extremely difficult to sanitise age from a resume.

Let's say the candidate got a BS in EECS in 1987, they've had 6 jobs since graduating, and their first job was cost-optimising floppy disk drives.

Do you feed the ML system this data, which is clearly correlated with age? Do you keep the number of jobs but delete the duration, responding to 6-jobs-in-3-years the same as 6-jobs-in-30-years? How do you sanitise the age-correlated data out of job descriptions naming old tech or defunct employers? Do you delete all but the last 5 years of their employment history, assigning no value to experience beyond that?

Re: A brief history and future of credit scores

#53

Post by dsr_ »

maccard wrote: post #47
ryeights wrote: post #21

Earlier quoted context omitted.

Thanks, economist.com is terrible

Meta comment but why do you say this yet read their content?

Suppose you think the average article in The Economist is terrible. One might then conclude that it's not worth reading in general. But HN acts as a recommendation, and that might be a strong enough signal to interest you in reading this particular article.

But The Economist has an unfriendly layout or puts up a paywall or blares ads at you, so you flip into reader mode as soon as you see the article. You retain an impression that The Economist is terrible, while still reading this article.

Does that help resolve your cognitive dissonance?

Re: A brief history and future of credit scores

#54

Post by dahfizz »

mindslight wrote: post #31
dahfizz wrote: post #29

Earlier quoted context omitted.

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

Isn't that a bit like asking how many pedestrians you can hit and still keep your license? The right answer is to try for zero. The dilemma you're describing isn't due to the law itself, but rather because the difficulty of writing law results in only the absolute worst abuses being criminalized. The abstract safe answer is to not engage in group-based discrimination at all , regardless of it seeming quite lucrative…

> Isn't that a bit like asking how many pedestrians you can hit and still keep your license?

This is nothing like that. Hitting or not hitting a pedestrian is binary, and there is a clear way to determine fault. We often call car collisions "accidents", but they aren't. Someone did something wrong to cause a collision.

The issue here is that you can be discriminatory "by accident". You can be 100% race blind and have the best intentions in the world and work very hard to be equitable but still have the slightest bias due to the nature of the data.

In fact, when poverty is correlated with race so strongly, I would argue it's impossible for a bank not to accidentally discriminate. A bank isn't going to lend you someone who is unlikely to pay it back - there's nothing racist about that. But they have had a disparate impact.

It's a very sticky situation.

Re: A brief history and future of credit scores

#56

Post by brighter2morrow »

teekert wrote: post #49
maccard wrote: post #48

Earlier quoted context omitted.

It's not really a US only thing - based on your username I'm assuming Dutch? - a quick Google says that there was 6 billion euro in new consumer credit in the Netherlands last year, and it's been much more than that even in the past decade. People in the Netherlands seem jsut as likely to buy cars on finance and rack up credit card debt as everyone else.

Oh.. must be my bubble then :)

Also, due to massive migrations the distinctions between countries are disappearing. So traditional Dutch may behave exactly as you say, but if immigrants from the same source go to different countries (like Netherlands and US) but act similarly then they will push these economic metrics towards each other across their host countries.

Re: A brief history and future of credit scores

#57

Post by AnthonyMouse »

jcranmer wrote: post #33
dahfizz wrote: post #29

Earlier quoted context omitted.

This gets at the heart of one of the issues with these discrimination laws. What if, all else being equal, people from zipcode A are statistically much more likely to default than those from zipcode B? Do financial firms have to pretend like they don't know that fact? How removed from race does information have to be in order to be considered by a financial firm?

> How removed from race does information have to be in order to be considered by a financial firm? In general, the standard is "disparate impact"--if you accepted 80% of all white applicants but only 20% of black applicants, then you're probably liable for racial discrimination even if you were completely race-blind.

> In general, the standard is "disparate impact"

That standard is unworkable. What happens when an idealistic charity goes into a homeless shelter in a black neighborhood with a high rate of drug use and helps them all fill out mortgage applications, so that 80% of the lender's white applicants are married with stable middle class jobs and 80% of the black applicants are single, homeless, unemployed and suffering from drug addiction? The institution evaluating the applications may not even be aware of why their applicant population skews that way.

Re: A brief history and future of credit scores

#58

Post by AnthonyMouse »

tareqak wrote: post #46
tareqak wrote: post #40

Earlier quoted context omitted.

From my reading of that article, I think the recruiting tool was fed resumés and a data point saying whether or not the corresponding candidate was hired or not. As a result, the tool not only developed a bias against women, but was effectively evidence that there was bias against women in the original hire / not hire decisions.

I just missed the deadline to edit my post, so I am replying to myself. Looking at the parent comment again, I seemed to have just restated it without adding anything new of my own. I meant to add that my reasoning for why Amazon pulled development of this tool was not just because the tool’s bias, but also because that the existence of the tool and its associated training data could open up Amazon to litigation clai…

It's interesting that nobody even bothered to check whether the bias was illicit. They found something that sounds bad and the immediate response is "OMG bad PR, pull emergency shutdown."

They just assume that "women's" is coding for female candidates and not something more specific, like gender-segregated activities that may legitimately produce lower quality candidates than the equivalent integrated activities that exposed the student/candidate to a more diverse cohort population. Probably also doesn't help that some of the biggest gender-segregated institutions are penal in nature, i.e. "reform school for troubled girls" or "women's correctional facility."

Did anybody even check whether it also penalizes words like "boys" and "gentlemen's"?

Re: A brief history and future of credit scores

#59

Post by jcranmer »

AnthonyMouse wrote: post #57
jcranmer wrote: post #33

Earlier quoted context omitted.

> How removed from race does information have to be in order to be considered by a financial firm? In general, the standard is "disparate impact"--if you accepted 80% of all white applicants but only 20% of black applicants, then you're probably liable for racial discrimination even if you were completely race-blind.

> In general, the standard is "disparate impact" That standard is unworkable. What happens when an idealistic charity goes into a homeless shelter in a black neighborhood with a high rate of drug use and helps them all fill out mortgage applications, so that 80% of the lender's white applicants are married with stable middle class jobs and 80% of the black applicants are single, homeless, unemployed and suffering fro…

There is a legal defense, where you can argue that the disparate impact is caused by practical business need instead of by implicit bias. An example (using gender instead of race) is that women have less upper body strength than men, on average, so they are far less likely to meet job requirements that require being able to carry 100 pounds of equipment on their backs.

Even then, however, you still have to demonstrate in your defense that the requirements are actually justified and not a backdoor proxy. In the example I gave earlier, you have to demonstrate that employees actually need to carry 100 pounds of equipment on their backs, and furthermore that there is no workable alternative to that requirement. If I were a legal compliance officer, machine learning for applicant screening would scare me, because I would have a hard time arguing in court that the results constituted a legitimate business need and not racism-by-proxy, especially if the plaintiff found that some of the metrics highly correlated to race.

Re: A brief history and future of credit scores

#60

Post by diminoten »

throwawaywego wrote: post #36
diminoten wrote: post #3

The key here being that non-financial data isn't actually useful in predicting ability to repay loans. I'm sure it'd be used somehow by modern financial institutions if it were predictive.

> The key here being that non-financial data isn't actually useful in predicting ability to repay loans. Sure it is. Even religion correlates with loan risk. (Could be a proxy for social status and people from your tribe helping you out when you can't pay back). I could probably get a predictive model better than random guessing by mining your HackerNews comments or Facebook likes. > I'm sure it'd be used somehow by…

Cool, I look forward to your evidence and data that you're using to contradict the story in the submission, because the very article states that if you tried to pitch a loan approval algorithm with any of the things you suggested in a modern bank, you'd be laughed out of the firm.
Post reply on HN